How can I animate a character arm/hand rig from recorded MediaPipe landmark data?

Questions
1

Hello,

I am working on a Three.js project where I want to animate a Mixamo character using MediaPipe landmark data that has been recorded and saved as JSON.

The landmark data is already available frame by frame, and my goal is to use it to animate the character’s arms, hands, and fingers during playback. I understand the general idea of converting landmark positions into bone rotations, but I have been struggling with the implementation.

I have already tried several retargeting approaches, including calculating rotations from landmark direction vectors and applying them to the Mixamo bones. However, the results are always poor. Depending on the method, the hands end up distorted, the fingers bend incorrectly, the rotations become unstable, or the overall motion simply does not resemble the original movement captured by MediaPipe.

At this point, I am starting to wonder if I am approaching the problem incorrectly. I would like to know how people typically handle the retargeting process when using MediaPipe landmark data to animate a rigged 3D character. Is there a standard workflow for this? Are there common mistakes or pitfalls that could explain why my results are consistently poor?

I found some older discussions on the forum involving similar projects, but the topics are now closed and I could not find a public explanation of the solutions that were used.

Any advice would be greatly appreciated.

Thank you.

2

Maybe GitHub - yeemachine/kalidokit: Blendshape and kinematics calculator for Mediapipe/Tensorflow.js Face, Eyes, Pose, and Finger tracking models. · GitHub ?

I’ve done this manually a long time ago, but it was painful. I had to make a hand tunes IK rig and filter the blazepose samples and do a lot of interpolation to make it look smooth.

Here’s a gemini slop test of kalidokit: