Attach additional Mesh to a Model with existing Animations?

Questions
, , ,
1

Given is a (GLB) Model (human body) with an Armature and Animations. Is it possible programmatically add a new Mesh (another GLB Model) to the animated Model and apply its animations to it?

Case 1: Attach new Model (e.g. sunglasses) to the existing Model (just translation + rotation + scale)
Case 2: Attach new Model (e.g. t-shirt) to the existing Model (all above + deformation)

Is this technically possible?

import React, { useRef, useEffect } from "react";
import { useGLTF, useAnimations } from "@react-three/drei";
export default function Model({ ...props }) {
  const groupRef= useRef();
  const { scene, animations } = useGLTF("/gltf/myGoodHuman.gltf", true);
  const { actions, mixer } = useAnimations(animations, group);
  useEffect(() => {
    actions.Animation.play();
  }, [mixer]);
  return <primitive ref={groupRef} object={scene} dispose={null} />;
}
useGLTF.preload("/gltf/myGoodHuman.gltf");

I know you can just apply the animations (also from other loaded models) to different models (like above applying animations to the groupRef). But how would I manage the missmatch of the position?

=> Load animation for Human1 and apply it to Human2 => (if similar mesh size/volume) no problem.

=>But load animation for Human1 and apply it to a Shirt => offset of the shirt so that anchor point is in the middle of the chest (simplified)??

2

you can just add stuff to any mesh or bone, but with a primitive it’s the same as in vanilla three, it’s dirty, because you’re mutating source data. if you can possible make it so that these models can be processed build time with gltfjsx i would suggest you do that.

otherwise you can declaratively write anything to anywhere with portals. a portal writes a jsx graph into a foreign, imperative node. the elements and components you render that way stay reactive.

on the bright side, it will be less hacky than vanilla because at least you get “nodes”, so you can target specific meshes without having to traverse. that is, if you’ve named them properly.

import { createPortal } from '@react-three/fiber'

export default function Model(props) {
  const { nodes, scene, animations } = useGLTF("/gltf/myGoodHuman.gltf")
  const { actions, mixer } = useAnimations(animations, scene)
  useEffect(() => void actions.Animation.play(), [actions])
  return (
    <>
      <primitive object={scene} />
      {createPortal(<SunGlasses />, nodes.head)}
    </>
  )
}

PS, i’ve cleaned the code above up a little. it was too verbose. also, bad bad idea to use gltf on the web. open your shell and type: npx gltfjsx myGoodHuman.gltf --transform. even if you can’t use the jsx, it will compress the hell out of that model and form a single myGoodHuman-transformed.glb.

3

Is createPortal doing the same as I would continuously get the position of the head and apply it to the sunglasses in useFrame?

Thanks for the hint with the --transform tag for gltfjsx.

But I assume this solution wouldn’t apply also the deformation through the armature bones to the attached mesh, would it?

4

useFrame? it mounts a declarative graph into a foreign object3d. that’s all there is to it. just like <Canvas> mounts its contents into its internal scene. you normally use it for effect composers and off-buffers.

const [scene] = useState(() => new THREE.Scene())
return createPortal(children, scene)

but you can also use it in your case, to add declarative, reactive code into some node that you got from your gltfloader. better than traversing and messing with add/remove, because that will strip all interactivity, and all reactivity.

But I assume this solution wouldn’t apply also the deformation through the armature

gltfjsx does nothing to the model, it just compresses it. if your model was 100mb before now it will be 1mb. but it will work the same way, include the same animations.

5

gltfjsx does nothing to the model, it just compresses it. if your model was 100mb before now it will be 1mb. but it will work the same way, include the same animations.

You got me wrong here, I meant that the gltf mesh is not getting deformed (because of the bones) when using createPortal to attach the one mesh to the other mesh.

useFrame… the animation loop, to apply the position of meshA to meshB on each frame.

6

keyframes move the actual bones and meshes. if you have stuff parented to that it must move along. although i have never tried it, but it seems like it should.

7 
	<group ref={group} {...props} dispose={null}>
		<primitive scale={[3, 3, 3]} position={[0, -3, 0]} object={nodes.RootNode}>
			<primitive object={hairs.nodes.RootNode} />
		</primitive>
    </group>

Translation (position) is applied, but no rotation:

image
image1034×724 44.8 KB

Now trying it with createPortal(hairs, head):

	<group ref={group} {...props} dispose={null}>
		<primitive scale={[3, 3, 3]} position={[0, -3, 0]} object={nodes.RootNode} />

		{createPortal(
			<primitive
				// scale={[3, 3, 3]}
				position={[0, -1.61, 0.01]}
				object={hairs.nodes.RootNode}
			/>,
			nodes.Head
		)}
  </group>

image
image1716×843 70.8 KB

Now Position and Rotation are applied, but the Animation is broken (notice the T-Shape of the body; a position or scale offset is happening on the animations rigg; means there is some movement from the animation, but with offset) and I had to adjust the position={[0, -1.61, 0.01]} with those magic numbers (I heavily assume the anchor point of the hair and the head are at the same point).

Conclusion: createPortal works somehow with position and rotation, but needs adjustment and the animation breaks.

Am I missing something?

8

i have no understanding of how animation works in threejs, i assume it needs the exact order of the scene to function. what you can also do is have the object outside, aside the skinned mesh, but not parented to it, and just apply the worldMatrix of the animated object (in a useFrame), i think that would be the cleanest solution in this case. since this is and needs math i wouldn’t be able to help with this, but i’m sure it will work.