How to assign custom data (position, rotation in x,y,z) to my skeleton?

Questions
1

I’m doing a project that can livestream from Rokoko Studio to a website using Three.js. This is an example of a frame from livestreaming feature

{
  "version": "3,0",
  "fps": 30.0,
  "scene": {
    "timestamp": 66.196785,
    "actors": [
      {
        "name": "Actor1",
        "color": [217, 119, 150],
        "meta": {
          "hasGloves": false,
          "hasLeftGlove": false,
          "hasRightGlove": false,
          "hasBody": false,
          "hasFace": true
        },
        "dimensions": { "totalHeight": 1.65, "hipHeight": 0.8762712 },
        "body": {
          "hip": {
            "position": { "x": 0.0, "y": 0.8762712, "z": 1.16682564e-7 },
            "rotation": { "x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0 }
          },
          "spine": {
            "position": {
              "x": -2.90140463e-7,
              "y": 0.9553583,
              "z": -0.0368060172
            },
            "rotation": { "x": 0.0, "y": 0.0, "z": 1.0, "w": 1.53080857e-17 }
          },
          "chest": {
            "position": {
              "x": 4.10987866e-8,
              "y": 1.21645176,
              "z": -0.0368061773
            },
            "rotation": { "x": 0.0, "y": 0.0, "z": 1.0, "w": 6.123234e-17 }
          },
          "neck": {
            "position": { "x": 8.006572e-8, "y": 1.40536952, "z": -0.03680656 },
            "rotation": { "x": 0.0, "y": 0.0, "z": 1.0, "w": 6.123234e-17 }
          },
          "head": {
            "position": {
              "x": 1.86135765e-7,
              "y": 1.50776827,
              "z": -0.0368065275
            },
            "rotation": { "x": 0.0, "y": 0.0, "z": 1.0, "w": 6.123234e-17 }
          },
          "leftShoulder": {
            "position": {
              "x": -0.0656277761,
              "y": 1.33240747,
              "z": -0.01895734
            },
            "rotation": {
              "x": -0.09229595,
              "y": 0.09229595,
              "z": 0.7010574,
              "w": -0.7010574
            }
          },
          "leftUpperArm": {
            "position": { "x": -0.1868263, "y": 1.33240736, "z": -0.051432386 },
            "rotation": {
              "x": 0.0273925271,
              "y": 0.6751999,
              "z": 0.0043296325,
              "w": 0.7371134
            }
          },
          "leftLowerArm": {
            "position": { "x": -0.194273189, "y": 1.089476, "z": -0.06267956 },
            "rotation": {
              "x": -0.00208534719,
              "y": 0.6757522,
              "z": -0.0278269239,
              "w": 0.736600757
            }
          },
          "leftHand": {
            "position": {
              "x": -0.204501092,
              "y": 0.82197696,
              "z": -0.05178108
            },
            "rotation": {
              "x": -0.00208534743,
              "y": 0.6757523,
              "z": -0.02782693,
              "w": 0.7366008
            }
          },
          "rightShoulder": {
            "position": {
              "x": 0.06562792,
              "y": 1.33240747,
              "z": -0.0189574659
            },
            "rotation": {
              "x": 0.09229595,
              "y": 0.09229595,
              "z": 0.7010574,
              "w": 0.7010574
            }
          },
          "rightUpperArm": {
            "position": { "x": 0.1868265, "y": 1.33240736, "z": -0.0514324121 },
            "rotation": {
              "x": 0.0273930784,
              "y": -0.6751999,
              "z": -0.00432998128,
              "w": 0.7371133
            }
          },
          "rightLowerArm": {
            "position": { "x": 0.194273248, "y": 1.08947587, "z": -0.06268048 },
            "rotation": {
              "x": -0.00208480051,
              "y": -0.675752163,
              "z": 0.02782657,
              "w": 0.736600637
            }
          },
          "rightHand": {
            "position": {
              "x": 0.204500869,
              "y": 0.821977139,
              "z": -0.0517822951
            },
            "rotation": {
              "x": -0.00208480051,
              "y": -0.675752163,
              "z": 0.0278265718,
              "w": 0.736600637
            }
          },
          "leftUpLeg": {
            "position": {
              "x": -0.0755084753,
              "y": 0.8762712,
              "z": 1.16682571e-7
            },
            "rotation": {
              "x": -0.00617059227,
              "y": 0.7070798,
              "z": 0.00617059227,
              "w": 0.7070798
            }
          },
          "leftLeg": {
            "position": {
              "x": -0.06848007,
              "y": 0.473620772,
              "z": 2.81151017e-7
            },
            "rotation": {
              "x": 0.02467767,
              "y": 0.706676,
              "z": 0.03700711,
              "w": 0.7061377
            }
          },
          "leftFoot": {
            "position": {
              "x": -0.0620454773,
              "y": 0.07156405,
              "z": -0.0351794548
            },
            "rotation": {
              "x": 0.0308435876,
              "y": 0.7064339,
              "z": -0.7064339,
              "w": 0.0308435466
            }
          },
          "leftToe": {
            "position": {
              "x": -0.07274504,
              "y": 0.0133998618,
              "z": 0.08711898
            },
            "rotation": {
              "x": 0.03084353,
              "y": 0.7064341,
              "z": -0.7064337,
              "w": 0.0308436137
            }
          },
          "leftToeEnd": {
            "position": { "x": 0.0, "y": 0.0, "z": 0.0 },
            "rotation": { "x": 0.0, "y": 0.0, "z": 0.0, "w": 0.0 }
          },
          "rightUpLeg": {
            "position": {
              "x": 0.0755084753,
              "y": 0.8762712,
              "z": 1.14460015e-7
            },
            "rotation": {
              "x": -0.00617059227,
              "y": -0.7070798,
              "z": -0.00617059227,
              "w": 0.7070798
            }
          },
          "rightLeg": {
            "position": {
              "x": 0.06848014,
              "y": 0.473620653,
              "z": 2.74483284e-7
            },
            "rotation": {
              "x": 0.02467767,
              "y": -0.706676,
              "z": -0.03700711,
              "w": 0.7061377
            }
          },
          "rightFoot": {
            "position": {
              "x": 0.0620454475,
              "y": 0.07156393,
              "z": -0.0351794958
            },
            "rotation": {
              "x": -0.0308435876,
              "y": 0.7064339,
              "z": -0.7064339,
              "w": -0.0308435466
            }
          },
          "rightToe": {
            "position": {
              "x": 0.0727451444,
              "y": 0.0133996531,
              "z": 0.08711893
            },
            "rotation": {
              "x": -0.0308435839,
              "y": 0.7064339,
              "z": -0.7064339,
              "w": -0.0308435522
            }
          },
          "rightToeEnd": {
            "position": { "x": 0.0, "y": 0.0, "z": 0.0 },
            "rotation": { "x": 0.0, "y": 0.0, "z": 0.0, "w": 0.0 }
          }
        },
        "face": {
          "mouthFunnel": 0.432980448,
          "mouthLowerDownRight": 0.7135263,
          "browDownLeft": 0.0,
          "jawLeft": 1.28944945,
          "jawOpen": 0.896597564,
          "mouthDimpleRight": 0.606104434,
          "mouthDimpleLeft": 0.6579531,
          "eyeLookDownLeft": 34.170063,
          "eyeLookDownRight": 34.0821266,
          "eyeLookInLeft": 1.43803966,
          "cheekSquintRight": 1.68572855,
          "mouthClose": 0.9909267,
          "mouthSmileLeft": 0.0,
          "mouthSmileRight": 0.0,
          "mouthPressRight": 2.44213271,
          "mouthLeft": 0.0,
          "mouthLowerDownLeft": 0.717432559,
          "mouthRight": 0.444443226,
          "mouthPressLeft": 2.44213271,
          "mouthRollUpper": 0.712731361,
          "mouthPucker": 4.91015625,
          "mouthRollLower": 1.02788866,
          "mouthShrugUpper": 4.1103754,
          "mouthShrugLower": 5.51735163,
          "mouthUpperUpRight": 0.8484512,
          "mouthStretchRight": 2.33507419,
          "mouthStretchLeft": 2.432367,
          "browInnerUp": 2.07726169,
          "browOuterUpLeft": 0.988449752,
          "browDownRight": 0.0,
          "browOuterUpRight": 0.9503638,
          "cheekSquintLeft": 1.527707,
          "eyeLookUpRight": 0.0,
          "cheekPuff": 1.52727532,
          "eyeLookOutLeft": 0.0,
          "eyeLookOutRight": 0.0,
          "eyeLookInRight": 10.5699549,
          "eyeWideLeft": 0.0,
          "jawForward": 0.421749979,
          "eyeWideRight": 0.0,
          "mouthFrownLeft": 6.059868,
          "mouthFrownRight": 4.60501575,
          "eyeSquintRight": 1.645871,
          "eyeLookUpLeft": 0.0,
          "mouthUpperUpLeft": 0.860169947,
          "eyeSquintLeft": 1.645871,
          "jawRight": 0.0,
          "noseSneerLeft": 5.4296875,
          "noseSneerRight": 4.3125,
          "tongueOut": 1.10855645e-8,
          "eyeBlinkLeft": 1.78045511,
          "eyeBlinkRight": 1.82324219
        }
      }
    ],
    "props": [],
    "characters": []
  }
}

If I want to assign these custom values to the each node, how should I approach it?

image
image741×501 81.1 KB

image
image900×518 42.5 KB

I assigned only the arms motions for above pictures. I can access each node already, but when I .position and .rotation.set, it doens’t work as it seems to be.

2

Did you modify the local or the world positions of the nodes?

3

I’m not sure. But from what I guess, it is local? Below is the code that I used to assign those new values.

export function updateBoneData(jsonData, model) {
  model = nodeTraverse(model, "Hips")

  const boneNamesToUpdate = [
    "RightShoulder",
    "RightArm",
    "RightForeArm",
    "RightHand",
    "LeftShoulder",
    "LeftArm",
    "LeftForeArm",
    "LeftHand",
  ]

  model.traverse((node) => {
    if (node instanceof THREE.Bone && boneNamesToUpdate.includes(node.name)) {
      let mappedName = mappedPart(node.name)
      if (mappedName && jsonData.scene.actors[0].body[mappedName]) {
        if (!node.getObjectByName("axesHelper")) {   // Adding AxesHelper to the bone (only once)
          const axesHelper = new THREE.AxesHelper(10)
          axesHelper.name = "axesHelper"
          node.add(axesHelper) // This is for axes 
        }
        const boneData = jsonData.scene.actors[0].body[mappedName]
        node.position.set(
          boneData.position.x,
          boneData.position.y,
          boneData.position.z
        )
        node.quaternion.set(
          boneData.rotation.x,
          boneData.rotation.y,
          boneData.rotation.z,
          boneData.rotation.w
        )
      } else {
        console.error(`No data found for bone: ${node.name}`)
      }
    }
  })
}

I did a search. And look like I need to do something with matrix4 or matrix4World? This is so challenging for me as a newbie.

4

Explanation

So there are actually 2 type of transformations:

  1. Local Transformation
  2. Global Transformation

You can better understand the difference between the two by watching this video, I also watched it to clear my concept:

Local Transformation vs Global Transformation

Although this explanation is in Unity but the logic is same for all graphic libraries so it also applies on three.js

Observation

Maybe currently what you are doing is changing the local position that’s why you are getting the weird transformations, modifying the world position might do the trick.

Judging from your code I assume you are using React to create your scene, i would suggest you to use React-Three-Fiber for this and then convert your model into a component using gltfjsx that would allow you to modify the nodes easily without traversing through each.

Is it possible that you can share your model if it’s not under companies TOS?

5

Oh, Thanks a lot for your explanation. I really appreciated for your answer. I’ve been suffering making it works for days. I’ll try my best to understand it and go for react three fiber. And If I have a further question, Can I update on this thread?
test.gltf (138.2 KB)
test.bin (4.9 MB)
The model is from Rokoko Studio. I think it’s called Rokoko Newton.

6

Sure, when you’ll update I’ll get the notification, if you need further assitance here’s my discord ashhad0001

Give me some time and I’ll look into the model :+1: