I have got a web app going using Google MediaPipe to animate a character (move the head, nothing fancy). It follows the original video pretty well. But when I turn it into an animation clip (which also involves moving from Euler to Quaternions), it’s been offset a bit.
Are there good examples to create a animation clip? Maybe I am missing a step - browsing the three.js code I noticed comments about a “reference track” for example.
Spent a few hours trying all sorts of adding and removing offsets, but cannot get it to come out right. (Note: I am putting 10% of Media pipe head rotations into the spine, 20% into the neck, 70% into the head. If you watch the video, playback creates a slouch, and I cannot work out why.)
Unfortunately the code is pretty complicated - intertwined with audio recording and the rest of the app. But I found it hard to find examples how to create an animation clip. So my question is are there any good examples around to follow. Not really expecting anyone to solve this from the video alone.
Here is the core of the bit of code creating the animation clip. I use the same algorithm to convert mediapipe output for the main view, which works correctly.
// Find the bones for rotations
const headName = findHead(this._model)
const neckName = findNeck(this._model)
const spineName = findSpine(this._model)
var headWeight = TOTAL_WEIGHT
if (neckName) headWeight -= NECK_WEIGHT
if (spineName) headWeight -= SPINE_WEIGHT
const makeTrack = (boneName, offset, weight) => {
if (boneName) {
const values = []
for (const r of this._keyframeRotations) {
values.push(offset.x + r.x * weight)
values.push(offset.y + r.y * weight)
values.push(offset.z + r.z * weight)
values.push(1)
}
tracks.push(new QuaternionKeyframeTrack(
`${boneName}.quaternion`,
this._keyframeTimes,
values
))
}
}
makeTrack(headName, this._offsetHeadRotation, headWeight)
makeTrack(neckName, this._offsetNeckRotation, NECK_WEIGHT)
makeTrack(spineName, this._offsetSpineRotation, SPINE_WEIGHT)
// Finally assemble the animation clip.
const animationClip = new AnimationClip(this._clipName, -1, tracks)