Examples of mimcing playback as a recorded animation clip?

Questions
1

I have got a web app going using Google MediaPipe to animate a character (move the head, nothing fancy). It follows the original video pretty well. But when I turn it into an animation clip (which also involves moving from Euler to Quaternions), it’s been offset a bit.

Are there good examples to create a animation clip? Maybe I am missing a step - browsing the three.js code I noticed comments about a “reference track” for example.

Spent a few hours trying all sorts of adding and removing offsets, but cannot get it to come out right. (Note: I am putting 10% of Media pipe head rotations into the spine, 20% into the neck, 70% into the head. If you watch the video, playback creates a slouch, and I cannot work out why.)

Unfortunately the code is pretty complicated - intertwined with audio recording and the rest of the app. But I found it hard to find examples how to create an animation clip. So my question is are there any good examples around to follow. Not really expecting anyone to solve this from the video alone.

Here is the core of the bit of code creating the animation clip. I use the same algorithm to convert mediapipe output for the main view, which works correctly.

    // Find the bones for rotations
    const headName = findHead(this._model)
    const neckName = findNeck(this._model)
    const spineName = findSpine(this._model)

    var headWeight = TOTAL_WEIGHT
    if (neckName) headWeight -= NECK_WEIGHT
    if (spineName) headWeight -= SPINE_WEIGHT

    const makeTrack = (boneName, offset, weight) => {

      if (boneName) {

        const values = []
        for (const r of this._keyframeRotations) {
          values.push(offset.x + r.x * weight)
          values.push(offset.y + r.y * weight)
          values.push(offset.z + r.z * weight)
          values.push(1)
        }

        tracks.push(new QuaternionKeyframeTrack(
          `${boneName}.quaternion`,
          this._keyframeTimes,
          values
        ))
      }
    }

    makeTrack(headName, this._offsetHeadRotation, headWeight)
    makeTrack(neckName, this._offsetNeckRotation, NECK_WEIGHT)
    makeTrack(spineName, this._offsetSpineRotation, SPINE_WEIGHT)

    // Finally assemble the animation clip.
    const animationClip = new AnimationClip(this._clipName, -1, tracks)
2

Not sure off the top of my head, but animation actions have a .weight associated with them, and if you have multiple animation clips playing with non-zero weights, they will be additive, so perhaps that accounts for the difference in behavior you’re seeing?

3

Thanks for the suggestion. I will double check the code. I am only intending one animation clip to be playing, but I use the same character flipping between recording (controlling the character from MediaPipe) then playback (playing back the recording). I will double check to make sure the two are not colliding somehow.

I was going to have a look for a “scene recorder” if one existed as well. I could then compare the recording it makes to the one I generate directly from code, to see what is different.

4

I tracked down the problem in the end - just recording here for future reference. It turned out related to converting the order of Euler angle x/y/z to Quaternion, which was only done generating the animation clip. (I could not get Euler’s working in animation clips then came across a comment that Quaternions should be used anyway.) Once I fixed that, the strange offset problem went away.

1 Like