Attach additional Mesh to a Model with existing Animations?

Questions
, , ,
1

Given is a (GLB) Model (human body) with an Armature and Animations. Is it possible programmatically add a new Mesh (another GLB Model) to the animated Model and apply its animations to it?

Case 1: Attach new Model (e.g. sunglasses) to the existing Model (just translation + rotation + scale)
Case 2: Attach new Model (e.g. t-shirt) to the existing Model (all above + deformation)

Is this technically possible?

import React, { useRef, useEffect } from "react";
import { useGLTF, useAnimations } from "@react-three/drei";
export default function Model({ ...props }) {
  const groupRef= useRef();
  const { scene, animations } = useGLTF("/gltf/myGoodHuman.gltf", true);
  const { actions, mixer } = useAnimations(animations, group);
  useEffect(() => {
    actions.Animation.play();
  }, [mixer]);
  return <primitive ref={groupRef} object={scene} dispose={null} />;
}
useGLTF.preload("/gltf/myGoodHuman.gltf");

I know you can just apply the animations (also from other loaded models) to different models (like above applying animations to the groupRef). But how would I manage the missmatch of the position?

=> Load animation for Human1 and apply it to Human2 => (if similar mesh size/volume) no problem.

=>But load animation for Human1 and apply it to a Shirt => offset of the shirt so that anchor point is in the middle of the chest (simplified)??

2

you can just add stuff to any mesh or bone, but with a primitive it’s the same as in vanilla three, it’s dirty, because you’re mutating source data. if you can possible make it so that these models can be processed build time with gltfjsx i would suggest you do that.

otherwise you can declaratively write anything to anywhere with portals. a portal writes a jsx graph into a foreign, imperative node. the elements and components you render that way stay reactive.

on the bright side, it will be less hacky than vanilla because at least you get “nodes”, so you can target specific meshes without having to traverse. that is, if you’ve named them properly.

import { createPortal } from '@react-three/fiber'

export default function Model(props) {
  const { nodes, scene, animations } = useGLTF("/gltf/myGoodHuman.gltf")
  const { actions, mixer } = useAnimations(animations, scene)
  useEffect(() => void actions.Animation.play(), [actions])
  return (
    <>
      <primitive object={scene} />
      {createPortal(<SunGlasses />, nodes.head)}
    </>
  )
}

PS, i’ve cleaned the code above up a little. it was too verbose. also, bad bad idea to use gltf on the web. open your shell and type: npx gltfjsx myGoodHuman.gltf --transform. even if you can’t use the jsx, it will compress the hell out of that model and form a single myGoodHuman-transformed.glb.

3

Is createPortal doing the same as I would continuously get the position of the head and apply it to the sunglasses in useFrame?

Thanks for the hint with the --transform tag for gltfjsx.

But I assume this solution wouldn’t apply also the deformation through the armature bones to the attached mesh, would it?

4

useFrame? it mounts a declarative graph into a foreign object3d. that’s all there is to it. just like <Canvas> mounts its contents into its internal scene. you normally use it for effect composers and off-buffers.

const [scene] = useState(() => new THREE.Scene())
return createPortal(children, scene)

but you can also use it in your case, to add declarative, reactive code into some node that you got from your gltfloader. better than traversing and messing with add/remove, because that will strip all interactivity, and all reactivity.

But I assume this solution wouldn’t apply also the deformation through the armature

gltfjsx does nothing to the model, it just compresses it. if your model was 100mb before now it will be 1mb. but it will work the same way, include the same animations.

5

gltfjsx does nothing to the model, it just compresses it. if your model was 100mb before now it will be 1mb. but it will work the same way, include the same animations.

You got me wrong here, I meant that the gltf mesh is not getting deformed (because of the bones) when using createPortal to attach the one mesh to the other mesh.

useFrame… the animation loop, to apply the position of meshA to meshB on each frame.

6

keyframes move the actual bones and meshes. if you have stuff parented to that it must move along. although i have never tried it, but it seems like it should.

7 
	<group ref={group} {...props} dispose={null}>
		<primitive scale={[3, 3, 3]} position={[0, -3, 0]} object={nodes.RootNode}>
			<primitive object={hairs.nodes.RootNode} />
		</primitive>
    </group>

Translation (position) is applied, but no rotation:

image
image1034×724 44.8 KB

Now trying it with createPortal(hairs, head):

	<group ref={group} {...props} dispose={null}>
		<primitive scale={[3, 3, 3]} position={[0, -3, 0]} object={nodes.RootNode} />

		{createPortal(
			<primitive
				// scale={[3, 3, 3]}
				position={[0, -1.61, 0.01]}
				object={hairs.nodes.RootNode}
			/>,
			nodes.Head
		)}
  </group>

image
image1716×843 70.8 KB

Now Position and Rotation are applied, but the Animation is broken (notice the T-Shape of the body; a position or scale offset is happening on the animations rigg; means there is some movement from the animation, but with offset) and I had to adjust the position={[0, -1.61, 0.01]} with those magic numbers (I heavily assume the anchor point of the hair and the head are at the same point).

Conclusion: createPortal works somehow with position and rotation, but needs adjustment and the animation breaks.

Am I missing something?

8

i have no understanding of how animation works in threejs, i assume it needs the exact order of the scene to function. what you can also do is have the object outside, aside the skinned mesh, but not parented to it, and just apply the worldMatrix of the animated object (in a useFrame), i think that would be the cleanest solution in this case. since this is and needs math i wouldn’t be able to help with this, but i’m sure it will work.

9

I applied this particular aproach but I’m facing an issue with resetting/ the position of my child mesh i.e the shoe to the same position as the bone(parent) position.

it looks like adjusting manually might adjust positions in the rendered scene but after exporting the same, the position is somewhere else.

Can you help me out by explaining the worldMatrix, .needsUpdate = true, localPosition, worldPosition and how to ensure that position (essentially sticks or gets updated) in r3f. I added the shoe like this

 {createPortal(
            <Shoes
              position={nodes.Web_Rig_01_004Ankle_L.position}
              quaternion={nodes.Web_Rig_01_004Ankle_L.quaternion}
              appliedShoes={props.appliedShoes}
            />,
            nodes.Web_Rig_01_004Ankle_L
          )}

to update position, I was checking the position and quartenion like so

  useEffect(()=>{
    group.current.position.copy(props.position)
    group.current.quaternion.copy(props.quaternion)
  },[])
  return (
    <group
      ref={group}
      {...props}
   >
      <mesh ref={group}
        geometry={nodes.Shoes2.geometry}
        material={materials.Shoes1}
        rotation={[Math.PI / 2, 0 , 0]}
        scale={0.01}
      />
     </group>

The following is how it looks after manually tweaking position in the jsx character (dont mind size of model now, I will compress them to 1 to 3 mb at most with animations)
model(11).gltf (6.1 MB)

image
image470×664 77.8 KB

but still when imported into blender, there is a rotation and position issue.

image
image394×516 42 KB

and l also noticed that the shoes has its own node(maybe root/pivot) and the rest of the character is separated.
image

Any ideas on how to proceed are appreciated