Apply cloth with no Skeleton (or Bones) to an Avatar with Mixamo Skeleton

CHOIX · June 8, 2023, 9:37pm

I know it’s an “easy” play when your cloth has already the same skeleton and bones as your rigged avatar! But what to do when your cloths does not have any skinnedmesh, skeleton nor bones - and you want to apply the same fbx animation on the avatar and the cloth?

My approach:

iterate over the bones of the skeleton of the skinnedmesh of the avatar (in my case I have the static list of the hierarchy - it’s the Mixamo standard)
create a basic t-shape pose skeleton with the same structure (Bone names) as the avatars
create a skinnedmesh from the geometry of the cloth
apply the skeleton on to the cloth skinned mesh
???
Win?! Lose!

Am I dreaming or is this task not feasible at all (which is the same as I would dream)?

import React, { Suspense, useEffect, useRef } from "react";
import { Canvas, useFrame } from "@react-three/fiber";
import { useGLTF, useFBX, OrbitControls} from "@react-three/drei";
import { GLTF } from "three-stdlib";
import { AnimationMixer, Object3D } from "three";
import * as THREE from "three";

interface ModelProps {
  url: string;
  forwardedRef: React.MutableRefObject<Object3D | undefined>;
}

const Model: React.FC<ModelProps> = ({ url, forwardedRef }) => {
  const gltf = useGLTF(url) as GLTF;
  console.log("my "+url, gltf.scene)
  return <primitive object={gltf.scene} ref={forwardedRef} />;
};

function SkinnedTshirt() {
  const { nodes } = useGLTF('/tshirt.glb')
  
  // define the skeleton structure based on the log
  const skeletonStructure = [
    {name: 'RootNode', parent: null, position: [0, 0, 0]},
    {name: 'Hips', parent: 'RootNode', position: [0, 0, 0]},
    {name: 'Spine', parent: 'Hips', position: [0, 1, 0]},
    {name: 'Spine1', parent: 'Spine', position: [0, 1, 0]},
    {name: 'Spine2', parent: 'Spine1', position: [0, 1, 0]},
    {name: 'Neck', parent: 'Spine2', position: [0, 1, 0]},
    {name: 'Head', parent: 'Neck', position: [0, 1, 0]},
    {name: 'HeadTop_End', parent: 'Head', position: [0, 1, 0]},
    {name: 'LeftShoulder', parent: 'Spine2', position: [-1, 0, 0]},
    {name: 'LeftArm', parent: 'LeftShoulder', position: [-1, 0, 0]},
    {name: 'LeftForeArm', parent: 'LeftArm', position: [-1, 0, 0]},
    {name: 'LeftHand', parent: 'LeftForeArm', position: [-1, 0, 0]},
    {name: 'RightShoulder', parent: 'Spine2', position: [1, 0, 0]},
    {name: 'RightArm', parent: 'RightShoulder', position: [1, 0, 0]},
    {name: 'RightForeArm', parent: 'RightArm', position: [1, 0, 0]},
    {name: 'RightHand', parent: 'RightForeArm', position: [1, 0, 0]},
    {name: 'LeftUpLeg', parent: 'Hips', position: [-0.5, -1, 0]},
    {name: 'LeftLeg', parent: 'LeftUpLeg', position: [0, -1, 0]},
    {name: 'LeftFoot', parent: 'LeftLeg', position: [0, -1, 0]},
    {name: 'LeftToeBase', parent: 'LeftFoot', position: [0, -1, 0]},
    {name: 'LeftToe_End', parent: 'LeftToeBase', position: [0, -1, 0]},
    {name: 'RightUpLeg', parent: 'Hips', position: [0.5, -1, 0]},
    {name: 'RightLeg', parent: 'RightUpLeg', position: [0, -1, 0]},
    {name: 'RightFoot', parent: 'RightLeg', position: [0, -1, 0]},
    {name: 'RightToeBase', parent: 'RightFoot', position: [0, -1, 0]},
    {name: 'RightToe_End', parent: 'RightToeBase', position: [0, -1, 0]},

    {name: 'LeftHandThumb1', parent: 'LeftHand', position: [-2, -0.5, 0]},
    {name: 'LeftHandThumb2', parent: 'LeftHandThumb1', position: [-1, 0, 0]},
    {name: 'LeftHandThumb3', parent: 'LeftHandThumb2', position: [-1, 0, 0]},
    {name: 'LeftHandThumb4', parent: 'LeftHandThumb3', position: [-1, 0, 0]},

    {name: 'LeftHandIndex1', parent: 'LeftHand', position: [-2, 0, 0]},
    {name: 'LeftHandIndex2', parent: 'LeftHandIndex1', position: [-1, 0, 0]},
    {name: 'LeftHandIndex3', parent: 'LeftHandIndex2', position: [-1, 0, 0]},
    {name: 'LeftHandIndex4', parent: 'LeftHandIndex3', position: [-1, 0, 0]},

    {name: 'LeftHandMiddle1', parent: 'LeftHand', position: [-2, 0.5, 0]},
    {name: 'LeftHandMiddle2', parent: 'LeftHandMiddle1', position: [-1, 0, 0]},
    {name: 'LeftHandMiddle3', parent: 'LeftHandMiddle2', position: [-1, 0, 0]},
    {name: 'LeftHandMiddle4', parent: 'LeftHandMiddle3', position: [-1, 0, 0]},

    {name: 'LeftHandRing1', parent: 'LeftHand', position: [-2, 1, 0]},
    {name: 'LeftHandRing2', parent: 'LeftHandRing1', position: [-1, 0, 0]},
    {name: 'LeftHandRing3', parent: 'LeftHandRing2', position: [-1, 0, 0]},
    {name: 'LeftHandRing4', parent: 'LeftHandRing3', position: [-1, 0, 0]},

    {name: 'LeftHandPinky1', parent: 'LeftHand', position: [-2, 1.5, 0]},
    {name: 'LeftHandPinky2', parent: 'LeftHandPinky1', position: [-1, 0, 0]},
    {name: 'LeftHandPinky3', parent: 'LeftHandPinky2', position: [-1, 0, 0]},
    {name: 'LeftHandPinky4', parent: 'LeftHandPinky3', position: [-1, 0, 0]},

    {name: 'RightHandThumb1', parent: 'RightHand', position: [2, -0.5, 0]},
    {name: 'RightHandThumb2', parent: 'RightHandThumb1', position: [1, 0, 0]},
    {name: 'RightHandThumb3', parent: 'RightHandThumb2', position: [1, 0, 0]},
    {name: 'RightHandThumb4', parent: 'RightHandThumb3', position: [1, 0, 0]},

    {name: 'RightHandIndex1', parent: 'RightHand', position: [2, 0, 0]},
    {name: 'RightHandIndex2', parent: 'RightHandIndex1', position: [1, 0, 0]},
    {name: 'RightHandIndex3', parent: 'RightHandIndex2', position: [1, 0, 0]},
    {name: 'RightHandIndex4', parent: 'RightHandIndex3', position: [1, 0, 0]},

    {name: 'RightHandMiddle1', parent: 'RightHand', position: [2, 0.5, 0]},
    {name: 'RightHandMiddle2', parent: 'RightHandMiddle1', position: [1, 0, 0]},
    {name: 'RightHandMiddle3', parent: 'RightHandMiddle2', position: [1, 0, 0]},
    {name: 'RightHandMiddle4', parent: 'RightHandMiddle3', position: [1, 0, 0]},

    {name: 'RightHandRing1', parent: 'RightHand', position: [2, 1, 0]},
    {name: 'RightHandRing2', parent: 'RightHandRing1', position: [1, 0, 0]},
    {name: 'RightHandRing3', parent: 'RightHandRing2', position: [1, 0, 0]},
    {name: 'RightHandRing4', parent: 'RightHandRing3', position: [1, 0, 0]},

    {name: 'RightHandPinky1', parent: 'RightHand', position: [2, 1.5, 0]},
    {name: 'RightHandPinky2', parent: 'RightHandPinky1', position: [1, 0, 0]},
    {name: 'RightHandPinky3', parent: 'RightHandPinky2', position: [1, 0, 0]},
    {name: 'RightHandPinky4', parent: 'RightHandPinky3', position: [1, 0, 0]},
];

  // create a bone for each bone in the structure
  const bones = {};
  skeletonStructure.forEach(({name, position}) => {
    const bone = new THREE.Bone();
    bone.position.set(...position);
    bones[name] = bone;
    bones[name].name = name;
});

  // set up the hierarchy of the bones
  skeletonStructure.forEach(({name, parent}) => {
    if (parent) {
      bones[parent].add(bones[name]);
    }
  });

  // create the skeleton
  const skeleton = new THREE.Skeleton(Object.values(bones));
  console.log("Shirt SKLETON", bones)
  // clone the geometry of the t-shirt and create a skinned mesh
  const tshirtGeometry = nodes.Cube.geometry.clone();
  const material = new THREE.MeshStandardMaterial({ skinning: true });
  const tshirt = new THREE.SkinnedMesh(tshirtGeometry, material);
  tshirt.scale.set(0.5,0.5,0.5)
  // bind the skeleton to the tshirt
  const rootBone = bones['Armature'];
  tshirt.add(rootBone);
  tshirt.bind(skeleton);

  // load the animation and apply it to the skinned mesh
  const { animations } = useFBX('/animation.fbx'); // adjust this path to your animation
  const mixer = new THREE.AnimationMixer(tshirt);
  const action = mixer.clipAction(animations[0]);
  action.play();
  console.log("my SHirt", tshirt)
  return <primitive object={tshirt} />;
}

const Scene: React.FC = () => {
  const avatarRef = useRef < Object3D > ();
  const tshirtRef = useRef < Object3D > ();
  const walkAnimation = useFBX("/animation.fbx");
  let mixer: AnimationMixer;

  useEffect(() => {
    if (avatarRef.current && tshirtRef.current) {
      // Parent the t-shirt to the avatar
      avatarRef.current.add(tshirtRef.current);
    }
  }, [avatarRef.current, tshirtRef.current]);

  useFrame((state, delta) => {
    if (mixer) mixer.update(delta);
  });

  useEffect(() => {
    if (avatarRef.current && walkAnimation.animations[0]) {
      mixer = new AnimationMixer(avatarRef.current);
      const action = mixer.clipAction(walkAnimation.animations[0]);
      action.play();
    }
  }, [avatarRef, walkAnimation]);

  return (
    <>
      <Model url="/avatar.glb" forwardedRef={avatarRef} />
      {/* <Model url="/tshirt.glb" forwardedRef={tshirtRef} /> */}
      <SkinnedTshirt></SkinnedTshirt>
    </>
  );
};

const App: React.FC = () => {
  return (
    <Canvas>
      <Suspense fallback={null}>
        <OrbitControls></OrbitControls>
        <ambientLight></ambientLight>
        <Scene />
      </Suspense>
    </Canvas>
  );
};

export default App;

Or is it that I could skipp all this and somehow just copy the skeleton from Avatars skinnedmesh into newly created skinnedmesh from the cloth?!

public replaceSkeleton( originalMesh: THREE.SkinnedMesh, newMesh: THREE.SkinnedMesh ): void {
    newMesh.skeleton = originalMesh.skeleton.clone();
    newMesh.bind( newMesh.skeleton );
  }

CHOIX · June 22, 2023, 4:47pm

If Mixamo can do it - it should be possible. No one ever tried this? (Beside Mixamo)

Lawrence3DPK · June 22, 2023, 5:02pm

instead of this, I think you’ll have to use SkeletonUtils.clone instead…

Topic		Replies	Views
Load a GLTF rigged mesh and add it to a skeleton of an existing rigged mesh Questions skinned-mesh , rigging	16	3180	February 4, 2022
Adding 3d model to already animated model Questions materials , geometry , animation , gltf-loader	3	1083	June 27, 2023
How to replace a part of skinnedMesh in R3F Questions geometry , gltf-loader	1	259	January 18, 2024
Add animation to GLB model Questions skeleton , bones , animation , mixamo , glb	1	3847	November 4, 2022
Replacing the skeleton in a GLTF rigged mesh with one from a different file Questions skeleton , gltf , skeletal-animation	3	1634	March 10, 2022

Apply cloth with no Skeleton (or Bones) to an Avatar with Mixamo Skeleton

Related topics