So I am working on a project, where I have this model(.glb)(wanted to make use vrm and maybe its inv kinemetics first, but Blender does not have a vrm exporter) and the posedetection model: PoseNet.
What PoseNet does when you feed it a video element, is return an array with the x, y position(and confidence) of your [keypoints] shoulders, elbows, wrists, hips, knees and ankles.
What I want to do
So what I want to do is, set(or what ever better way there is) the position and rotation of the bones in my model in a way so that it somewhat resembles my movement. So I am trying to figure out what the best way/mechanic is to move and rotate Bones for this model. Right now my only problem is to figure out a way to translate a x and y coordinate into a rotion and movement.
What I have thought about
I thought that maybe you could rotate the rotation or position of an object with a vector just like in unity, but could not get that to work with THREE.Vector2/3.
I thought I could use the Math.atan(delta.x/delta.y), but you get in the problem when you have a obtuse angle.
Sadly just transforming bones to the x,y cords are not enough eighter because the position of bones aren’t equal to the scene coordinates(0,0,0 for bones is not 0,0,0 for the scene), unless ofcourse there is a good way I have not figured out. There is also this problem when you would put, for example, your right arm on the left side of your body, with hardcoding the math.
I have been testing for a good metod by using mesh.getChildByName(“n”).rotation/position.x(.set)=k
Also you might notice that e.g. rotation.x=1 for arm1r is y mirrored for arm1l.
Maybe it could be done better if I would have had rigged my mesh better/smarter, I had to connect the shoulder and hip/leg1 bones because of transforms not being local/relaltive to the prev connected bone. Therefor I want to share my .blend file as well.
Map only containing rigged blend file
Blendmodel.zip (2.5 MB)
My folder with everything neccesary(I blieve)
question.zip (1.2 MB)
Keypoints returned btw
//var KeypointNames = [“nose”, “leftEye”, “rightEye”, “leftEar”, “rightEar”, “leftShoulder”, “rightShoulder”, “leftElbow”, “rightElbow”, “leftWrist”, “rightWrist”, “leftHip”, “rightHip”, “leftKnee”, “rightKnee”, “leftAnkle”, “rightAnkle”]
Names of the bones for easy hierarchy(for right just replace l with r)
hips{
hipsl{ leg1l{ leg2l }},
spine{ body{ neck, shoulderl{ arm1l{ arm2l }}}}
}
I want to thank everyone who helped or thought about helping or took the time to read my post, I’am and have put quite a lot of work in this(post and project) and I hope I will get helpfull answers. It is very late though(4am) and will be reading replies tomorrow(if I am that lucky).