How can I successfully mirror a SkinnedMesh that is used in a CCIKSolver?

Questions
,
1

I’m loading a glTF model of a right hand, and then loading it again (later I’ll just clone the already-loaded one) and attempting to flip it in order to create a left hand.

I know a mesh/Object3D can be flipped via applyMatrix4(), or by scaling one or more axes by -1. However this doesn’t flip the skeleton; also it makes the lighting really weird (maybe the normals need to be updated). SkinnedMesh.applyMatrix4() also does not flip both.

If I flip the skeleton alone by simply negating the z’s of the vertices (and leaving the rotations), I think I get what one would expect: the skinning pulls the thumb and other fingers to their opposite sides, which sort of twists the mesh. Code:

            theHand.traverse((child) => {
                if (child.isBone) {
                    child.position.z *= -1;
                }
            });

Result:

Screenshot 2024-07-20 at 5.43.58 PM
Screenshot 2024-07-20 at 5.43.58 PM796×818 62 KB

Here’s the right hand which looks normal, for comparison:

Screenshot 2024-07-20 at 5.44.08 PM
Screenshot 2024-07-20 at 5.44.08 PM814×742 71.9 KB

If I attempt to flip the mesh via theHand.applyMatrix4(new THREE.Matrix4().makeScale(1, 1, -1));, it causes the fingers (which are set up with CCDIKSolver) to flail all over, which I’m guessing indicates that the bones are not aligned with the mesh:

Screenshot 2024-07-20 at 9.20.35 PM
Screenshot 2024-07-20 at 9.20.35 PM1100×954 43.2 KB

So then I try flipping the vertices manually:

            theHand.traverse((child) => {
                if (child.isBone) {
                    child.position.z *= -1;
                } else if (child.isMesh) {
                    const vertices = child.geometry.attributes.position;
                    for (let i = 0; i < vertices.count; i += vertices.itemSize) {
                        vertices.array[i + 2] *= -1;
                    }
                }
            });

Result: The lower finger bones and the write seem in tact but the middle portions’ Z (left/right of the hand) seems to be way off:

Screenshot 2024-07-20 at 6.17.08 PM
Screenshot 2024-07-20 at 6.17.08 PM1324×1204 51.3 KB

I also tried various other things like flipping the rotation.x and rotation.y on each bone, but at this point I’m guessing. As another side note, going into Blender and doing “Mirror Global” on the model then exporting a separate left hand model, I get the flailing fingers.

(Also asked on StackOverflow, will sync updates from/to there: What is the proper way to Mirror a SkinnedMesh in three.js? - Stack Overflow )

2

Any chance you can provide a minimal demo? Are you using skeleton utils to clone the skeleton over?

:thinking: Doing this may require a few more slightly more complex operations such as reordering / reversing the skinWeights & skinIndices… The official docs on skinned mesh has a comprehensive bit of boiler code…

This is odd, mirroring skinned meshes can sometimes rely heavily on the original orientation, direction and position of the bind pose, have you tried different options for mirroring on other axes? In older versions of blender there was an armiture options sub menu that has an option for X axis mirroring…

Are the fingers flailing in blender if you run an animation or only when imported to three?

3

Hey, thanks for taking a look, Lawrence3DPK. I put together a minimal demo, tested with python3 -m http.server 8000, slightly too big for uploading here, so it’s on Google Drive: hands_minimal_demo.zip - Google Drive . Currently it’s trying to flip it in HandControls.prepareHandModel() by applying negative Z scale on the mesh and on each bone’s position (hand_demo.js:137).

So currently I’m just re-loading the model afresh instead of cloning, just to remove any possibility of cloning problems while trying to figure this out.

In Blender I did also try mirroring from the armiture options, but that does weird things with the bones. Uploaded that file:
right-hand-v001-jb2.blend (5.8 MB)

1 Like
4

This is just some additional info, which might be helpful.

In two of my projects (Mannequin and Virtual robotics) there is left-right symmetry of body parts. To support left-right symmetry of shapes and motions, I had to:

  • negate only some of the position coordinates
  • negate only some of the Euler angles

If I keep the same angles, or negate all angles, the motion breaks – the character moves like an out-of-control puppet doll. Both projects are not based on Three.js skeletons, but still the concept might be valid for skeletons.

1 Like
5

Hey, I just tried all 6 combinations of negating the three euler rotations of each bone, but still flailing in all cases, so I guess there’s another or an additional problem.

                    child.position.z *= -1;

                    child.rotation.x *= -1;
                    child.rotation.y *= -1;
                    // child.rotation.z *= -1;
6

Flipping the mesh plus only the root bone of the hand at least eliminates the flailing and gets us partially there (tried this because I guessed that the -1 scale transformation would get propogated to the root bone’s children). Now it’s upside down or something, and the normals seem reversed. Attached TransformControls to the root bone and tried rotating manually, but it rotates it in such weird ways that I could never get the hand to be in a reasonable state; plus it looks like the thumb might still be on the wrong side for left hand. So I’m not really sure if I’m closer to the solution or not.

            theHand.applyMatrix4(new THREE.Matrix4().makeScale(1, 1, -1));

            const rootBone = findBoneByName(skeleton.bones, `forearm${HAND_LETTER[this.handIndex]}`)
            rootBone.scale.x *= -1;

        theHand.rotation.y = Math.PI;
7

Managed to do this:

Attached is the modified file. See lines 157-162. I also found that positioning and rotation is very strange. I have no time to tackle with this. At least both hands look symmetrical, motions are also symmetrical, and coloring looks normal.

hands_demo.zip (3.3 KB)

1 Like
8

Any chance you can briefly summarise your workings here, very interesting stuff for future reference :slight_smile:

9

Did this:

if (this.handIndex === 0){ // for flipped hand
	theHand.rotation.set( -1, 0, 3.14, 'YXZ');
	theHand.position.set( 0, 20, -20);
	skeleton.bones[0].scale.x = -1; // flips motion
	skinnedMesh.scale.x = -1; // needed for good lighting (fixes normals)
}

The position and rotation of the hand are set by trial and error to make the mirror hand next to the original hand. Most likely somewhere in the structure there is a matrix, or other properties, that can be used to resolve this.

Finger motion is done this way:

leftHandControls.boneByName.thumb01R.rotation.x = Math.sin(time/200)*0.5-1;
leftHandControls.boneByName.f_index01R.rotation.x = Math.sin(time/200+0.5)*0.5+0.3;
leftHandControls.boneByName.f_middle01R.rotation.x = Math.sin(time/200+1)*0.5+0.3;
leftHandControls.boneByName.f_ring01R.rotation.x = Math.sin(time/200+1.5)*0.5+0.3;
leftHandControls.boneByName.f_pinky01R.rotation.x = Math.sin(time/200+2)*0.5+0.3;

rightHandControls.boneByName.thumb01R.rotation.x = Math.sin(time/200)*0.5-1;
rightHandControls.boneByName.f_index01R.rotation.x = Math.sin(time/200+0.5)*0.5+0.3;
rightHandControls.boneByName.f_middle01R.rotation.x = Math.sin(time/200+1)*0.5+0.3;
rightHandControls.boneByName.f_ring01R.rotation.x = Math.sin(time/200+1.5)*0.5+0.3;
rightHandControls.boneByName.f_pinky01R.rotation.x = Math.sin(time/200+2)*0.5+0.3;
1 Like
10

Thanks for taking a shot at this Pavel! However with the IK solver enabled, the left hand fingers still flail. I’ve noticed before that the flailing goes away when I disable the IK setup. So it appears to be something that the IK solver gets “confused” about with the skeleton+mesh setup I guess.