Need help figuring out bone rotations - please help!

Questions
, , ,
1

I have a rigged model setup my scene of a hand:

i1
i11179×697 141 KB

I also have position data of a new hand pose that I would like this model to take the form of:

i2

(^plotting of the position data [visually mirrored as well], I included the sample position data in the uploaded project files)

It should be self explanatory, but for the sake of clarity the key points in this position data represent where the joints between each bone are in 3D space.

The problem is (or at least I think) is that I cannot just move each joint in my rigged model to these points. I need to calculate the rotations from each key point to another and set the Bone Object rotations for each bone on my rigged model/skeleton to have it match the pose. And it makes it more difficult because the rotations are relative to bone previous to it, so once you calculate and set the rotation of the parent, it changes the coordinate system of the child.

So, basically I need an algorithm that takes in these key points for each finger and calculates and sets the correct rotations for each bone on my model to match the pose.

I have a very limited background in linear algebra, and I’ve tried so many different approaches to get this right from using basic trigonometry, axis angle rotations, converting direction vectors to Euler angles, calculating rotation matrices and basis transformations, messing with Quaternions, but I’m really struggling to get anywhere, and I’ve been stuck on this for over a week now, and I’ve tried posting on stackoverflow and gamedev and no one seems to be able to give me help or guidance.

Can someone please help me figure out how to this?

I attached my entire project if necessary. After installing the node packages, it should be immediately ready to go. I’m not asking anyone to do this for me, if anyone could at least help me with that math or any other help at all it would be much appreciated. This project is very important to me! Thank you for your time.

project.7z (296.7 KB)

another resource I found: obviously leap motion has a visualizer that’s similar to this. I can’t tell if these calculations in this code leapjs/threejs-bones.html at 5a89a25b5fd627ec0febeb7e8db77c6ca30f612a · leapmotion/leapjs · GitHub are a clue to the calculations needed to help me

2

I haven’t looked at your project, but are those points just points? If you manage to get the bones to look from one point to another, how will you determine the Roll value of that bone?

You can get the matrixWorld of an object/bone. It will have the world transforms, and from that you can retrieve the orientation of a child object/bone regardless of the orientation of its parent. Or you can get the matrix (not the matrixWorld), and it will represent the local transforms (relative to the parent).

You can also create those matrices for the bones by passing position/rotation/scale to a matrix4, and then applying that to the object/bone.

One thing you can do is move the bone into the scene object, so that it isn’t a child of anything, and apply the matrix to it so that it fits the world transforms… then you can Attach it to a parent bone essentially converting its world transforms into local transforms (attach is different from add).

So for simplicity sake, if you just focus on positions first…
Get world position of the points… set a bone position to each of those (make sure bones aren’t parented by moving them into the scene).
Then beginning with the bone that is the root bone, attach(don’t add) each bone in sequence. This will convert their world position into a local position (rotation and scale too).

Of course, you’ll also want to calculate the orientation as well, so you could point the bones at the child bone prior to attaching them all. But you’ll need to figure out how to roll them so that the outer and inner sides of the fingers are correct… You’d need more data from your points in order to do that though if your points are just positions.

Anyways, these are just some thoughts I had to get you thinking about things. If your initial points are just positions, I’m not sure if you can get an accurate pose.
anyways… maybe someone else has better ideas.

3

@prominent Those points are just points, yeah, if you mean 3D points (x, y, z). If youre asking if there are other things calculated like rotations, no

I think you have some good ideas, specifically detaching each bone, orienting it, and then reattaching it sounds like it might work

I did create a function that basically makes a bone look at a local point, so for example lets say the points for the index finger are:

knuckle (0, 0, 0) //p1
joint1 (0, 1, 1) // p2
joint2 (0, 1, 2) // p2
tip (0, 0, 2) //p3

visually (bottom axis is z and up axis is y):

i3

We use the function:

function boneLookAtLocal(bone, position) {
    bone.updateMatrixWorld()
    let direction = position.clone().normalize()
    let pitch = Math.asin(direction.y)// + bone.offset
    let yaw = Math.atan2(direction.x, direction.z); //Beware cos(pitch)==0, catch this exception!
    let roll = 0;
    bone.rotation.set(roll, yaw, pitch);

    console.log(roll, yaw, pitch);
    console.log(roll, THREE.MathUtils.radToDeg(yaw), THREE.MathUtils.radToDeg(pitch));
}

we use this function to make the index finger look at p2. This accurately sets the orientation of the first bone on the index finger:

        boneLookAtLocal(hand.index[0], p2)

image

Now moving on, we want to set the middle bone to an orientation where its just going forward.

As you can see by the axis helpers, the middle bone’s local coordinate system has tranformed, so when we call:

        let newDirection = new THREE.Vector3().subVectors(p3, p2) // gets forward direction vector = (0, 0, 1)
        boneLookAtLocal(hand.index[1], newDirection )

the bone does not point forward and instead points in the same direction because thats the direction of p3 on its local coordinate system:

i5

So taking part of your idea, if instead of trying to calculate a new direction vector on the middle bones local coordinate system that will point the bone forward in the direction we want, we detach the bone, orient it on the global access and the reattach it to where it was, will that work?

In case someone brings it up, I tried subtracting the offsets of the parent bone (yaw and pitch) when calling boneLookAtLocal() on the child bone to orient it correctly and that does not work for some reason

4

When you get the newDirection by subtracting p3 from p2, you can convert that into the local position of the bone…
You’ll need to first add the world position of the bone to the newDirection, so that the point represents a world position…
newDirection.add( hand.index[1].position );
hand.index[1].worldToLocal( newDirection );
Then you can do:
boneLookAtLocal(hand.index[1], newDirection );

That should make it point in the correct direction.

5

I tried that before too, this is the result:

i6

        boneLookAtLocal(hand.index[0], new THREE.Vector3(0, 1, 1)) // first rotation

        let middleBoneDirection = new THREE.Vector3(0, 0, 1)
        middleBoneDirection.add(hand.index[1].position)
        hand.index[1].worldToLocal(middleBoneDirection)
        boneLookAtLocal(hand.index[1], middleBoneDirection)

It should be pointing flat, forward towards the z axis but its angled upwards to the left instead?

6

Oh, I think you need to get world position of the hand.index[1].position (since the bone is a child)… You can create another temp vector and do…
hand.index[1].getWorldPosition( tempVector );
then add that to the newDirection…

so here’s the steps…

hand.index[1].getWorldPosition( tempVector );
newDirection.add( tempVector );
hand.index[1].worldToLocal( newDirection );
boneLookAtLocal(hand.index[1], newDirection );

Try that…

7

doesn’t work :frowning:

i7
i7635×532 146 KB 

        boneLookAtLocal(hand.index[0], new THREE.Vector3(0, 1, 1))
        let newDirection = new THREE.Vector3(0, 0, 1)
        let tempVector = new THREE.Vector3()
        hand.index[1].getWorldPosition( tempVector );
        newDirection.add( tempVector );
        hand.index[1].worldToLocal( newDirection );
        boneLookAtLocal(hand.index[1], newDirection );
8

Ok…
Try…

    boneLookAtLocal(hand.index[0], new THREE.Vector3(0, 1, 1))
    let newDirection = new THREE.Vector3(0, 0, 1)
    let tempVector = new THREE.Vector3()
    hand.index[1].getWorldPosition( tempVector );
    newDirection.add( tempVector );
    hand.index[1].parent.worldToLocal( newDirection );
    newDirection.sub( hand.index[1].position );
    boneLookAtLocal(hand.index[1], newDirection );

I think that’ll work, heh… it will convert the position to the parent’s local space, and then when subtract the bone’s position it will have the final position to look at locally.

9

i8

I dont know why this doesnt work either, that should work in my head

10

What does it look like with the visual indicators for the bone’s axis? Maybe your lookat function is rotating around the wrong axis? I know that objects in three.js look down the z axis, and so that axis would control the Roll I think. y would be Yaw, and x would be Pitch… at least I think so… Your function has them differently. Maybe you can try changing those?

11

i9
i9801×637 179 KB

is this what you mean by visual indicators, the axis helpers? And okay ill try setting the rotations as (pitch, yaw, roll) and other combinations see if that changes it

12

@prominent I think I found a different solution, attach the bone in question to the scene before rotating it to rotate it on the global access instead of local and then reattach the bone to its parent:

function boneLookAtLocal(bone, position) {
    bone.updateMatrixWorld()
    let direction = position.clone().normalize()
    let pitch = Math.asin(-direction.y)// + bone.offset
    let yaw = Math.atan2(direction.x, direction.z); //Beware cos(pitch)==0, catch this exception!
    let roll = Math.PI;
    bone.rotation.set(roll, yaw, pitch);
}

function boneLookAtWorld(bone, position) {
    scene.attach(bone)
    boneLookAtLocal(bone, position)
    bone.parent.attach(bone) 
}

boneLookAtWorld(hand.wrist, new THREE.Vector3(1, 1, 1))
boneLookAtWorld(hand.index[0], new THREE.Vector3(1, 1, 1))
boneLookAtWorld(hand.index[1], new THREE.Vector3(1, 0, 1))
boneLookAtWorld(hand.index[2], new THREE.Vector3(0, -1, 0))

yields:

i10.PNG
i10.PNG928×698 120 KB

which is how is exactly what I want! However, if we rotate the wrist right after those 4 calls in a different direction:

i11
i11864×612 168 KB

The wrist rotates without the rest of the finger. And if you noticed in the previous picture, the blue and green lines that make up the finger on the skeleton helper disappeared, so I’m thinking that when I call attach() it removes the bond from following the rotation of the parent.

Is there anyway to “re-astablish” that bond without resetting the fingers rotations? or anyway to capture the position it would be in and then move it there or?

13

hm, that’s interesting to know… I haven’t worked much with skinned meshes, so I’m not sure.

I was going to suggest maybe using the built in lookAt() function of the objects/bones instead of your function… to look at the position, to at least test and see if that points them in the correct direction( they might not have their roll correct though)…

I’m not sure how to rebind the mesh to the bone if attaching removes it.

14

Thats a good idea, let me try that as well. Thank you for you help so far!

15

@prominent in case it helps you one day, I figured it out!

The reason this is happening is because in:

function boneLookAtWorld(bone, v) {
    scene.attach(bone)
    boneLookAtLocal(bone, v)
    bone.parent.attach(bone)
}

when we attach the bone to scene, scene becomes the bone’s new parent. So after we perform the rotation via boneLookAtLocal() and reattach the bone, we aren’t reattaching the bone to the original parent! The bone is just being reattached to the scene which it is already attached to.

This is solved by changing boneLookAtWorld() to:

function boneLookAtWorld(bone, v) {
    const parent = bone.parent;
    scene.attach(bone)
    boneLookAtLocal(bone, v)
    parent.attach(bone)
}
16

oh yeah, that makes sense :slight_smile:
nice that you got it working!