Retargeting animation to Mixamo rig

Questions
1

I have a character that I had rigged by Mixamo, and I have a few animations in BVH-files from the Carnegie Mocap Library. I want to transfer the animation clips to this character.

I could not find any information anywhere how to do this in ThreeJS.

I did have a look at the SEA3D / BVH retargeting example, but that one seems to use the BVH skeleton instead of the rig the character already has. And I don’t want to end up with a lot of skeletons…

Anyone some hints how to get started?

1 Like
2

Hey all, no one any suggestions?

3

Did you ever find a resolution for this?

5

@Travis_Ealy this would only be possible if the BVH skeleton is identical to the Mixamo skeleton, which is unlikely.

6

As mentioned by the OP the repository once had an example that demonstrated the usage of SkeletonUtils.retarget(). It mapped an animated BVH skeleton to a different SEA3D skeleton.

https://rawcdn.githack.com/mrdoob/three.js/r105/examples/webgl_loader_sea3d_bvh_retarget.html

This seems to be the logic that is required for the OPs use case. It might be useful to replace the example with a new one using glTF and complete the documentation of SkeletonUtils.

3 Likes
7

@looeee thank for your reply. Do you think you can help with this. Let me know if we can connect.

8

Hi,

So I created a new rig. Same bones, I have to pull in the bvh animation dynamically so I saved the rig without the animations. I then pushed the animation into the array. However with the same bones Im getting the same result. Here’s a codebox:

9

Do you have anything working to demonstrates this? You say how this “seems to be the logic”, however it’s not. The SEA3d creates their own loader thats creates the mesh. bones and much more on load. This seems to be an issue that has been asked for years in the community but no one can get an answer.

10

Sorry, re-targeting is a complex topic and I have currently no time to dive into it. Somebody else has to help here…

1 Like
11

Thanks for getting back? Who else can I ask?

1 Like
12

Do you have to help me with this? I have a file set up and got pretty far. Just hung up on a few things? Mugen87 answer is pretty far off. Please let me know. Thanks

2 Likes
13

Hey do you found a solution for this? looks like the aniamtionclip names are different

1 Like
14

I am looking at this a bit as well. This is a better link to see the source code for that example:

  1. Bone name mapping - Convert the original skeleton bone names to the target bone names
  2. THREE.SkeletonUtils.getSkeletonOffsets - This seems to be getting the offsets (position, rotation, and maybe scale) of the original skeleton and the target skeleton
  3. THREE.SkeletonUtils.retarget( player, bvhSkeletonHelper, options ) - The retarget is being called in the animate() loop, bringing in the options

It would be nice if these examples were a bit more clean (bare bones), and also adding comments to the code to explain why things are being done. I might first try to run this example on version 105, then see if newer versions of three.js have this example broken.

15

This is the kind of question where it seems obvious but is actually devilishly complicated. Animation retargeting is a giant topic. Almost all attempts at automation of import etc. will break in one way or another, which is why there aren’t great tools for it, except for in 3d modellers like Blender/Maya, and Mixamo.
In a professional context you would usually work with an artist who knows how to retarget animations in a tool like blender, and/or use something like mixamo to retarget the animations, and then have the artist clean up the results.
Even when automated solutions sorta work, they usually require manual cleanup of the resulting animations.

Retargeting usually involves making an inverse kinematic rig of the target skeleton, and then using the source animation to drive the IK constraints, and then recording the resulting animation of the target.

Here’s some code to read about, perhaps it will help clarify the topic:

And some youtube content on the subject:

https://www.youtube.com/results?search_query=animation+retargeting

That all said… mixamo can be pretty effective, but its a whole process… not something you can just automate with a loader.

3 Likes
16

@manthrax -This does seem pretty difficult. I spent a bit of time taking the working example in version 105, bringing three.js up to version 160, converting to modules, then swapping the model/skeleton with a FBX mixamo model since the SEA3D importer kind of died.

I kind of have the retargeting working on the upper body. The “getSkeletonOffsets” function in the SkeletonUtils is what is doing most of the magic with reorienting bones from the original skeleton to the target skeleton. That function seems to have gotten dropped when the retargeting example was dropped as well. I am currently looking into how to deal with the lower extremities. There is an odd bone in the Carnegie Mellon skeleton with a “buttucks” bone that is nontypical for skeletons. I think that is what is throwing off targeting from fully working with the mixamo bone rig in this example.

Progress so far…

image
image1022×671 140 KB

I think the difficult thing with trying to make this simple to put in a library is that there are so many types of skeletons out there, and there are going to be a lot of variations. If something is added back to three.js that does retargeting, it might be better to add some building blocks to make this work like calculating the offsets for a bone, or show an example of how to do a bone mapping between a source bone and a target bone.

2 Likes
17

Yep. I’ve been there. Good times. :smiley:

fwiw, you can get a lot of mileage out of the .attach method… attach works differently than .add in that it preserves the world transform of the object…
https://threejs.org/docs/#api/en/core/Object3D.attach
so if you can get the skeletons lined up together in a bind pose, then .attach the source bones to the destination bones parent… the source bones .matrix will then be the transform difference between the bones.

2 Likes