I am trying to get the closest bone based on a Raycaster intersection, it works but I am not sure if it gets the closest one, I would appreciate some help, here is the function…
// Raycaster.
if(object instanceof THREE.SkinnedMesh) {
const skinnedMesh = object;
let fundBone;
if(raycaster){
const skinnedMeshIntersects = raycaster.intersectObject(skinnedMesh, true);
if(skinnedMeshIntersects.length > 0){
const localIntersectionPoint = skinnedMeshIntersects[0].point.clone();
let closestDistance = Number.MAX_VALUE;
for(let i = 0; i < skinnedMesh.skeleton.bones.length; i++) {
const bone = skinnedMesh.skeleton.bones[i];
// Transform the local intersection point to the bone's local space.
let localPoint = localIntersectionPoint.clone();
// Add small z offset.
const intersection = skinnedMeshIntersects[0];
localPoint = intersection.point.clone();
bone.worldToLocal(localPoint);
// Compute the distance from the bone's origin to the intersection point.
const distance = localPoint.length();
if (distance < closestDistance) {
closestDistance = distance;
result = {'boneName':bone.name, 'bone': bone}
}
}
}
}
Bones in threejs don’t have a length, and they don’t have a well defined “long” axis… you can probably easily find the nearest joint to a ray, but only if you know what axis the bone lies along, can you guess at what it’s endpoint is, by taking the position of one of its child bones, which works for things like… knees, but only for armatures that have well defined joints. If it doesn’t have a child bone… you don’t know if its a long Z axis bone or a Y axis bone… and some export formats/animations are exported with long Z and some with long Y … the skinned mesh code doesn’t care because it works either way…
but it limits our ability to think of bones as actual line segments with a start and end point.
So… long story short… as far as I know. there is no universal/robust way to identify a bone as a line segment, or a cylinder or anything useful. You can hack a solution that will work with a specific rig setup… and that works, but it won’t work for all cases.
For instance… some rigs have bones running along the spine, but then the shoulder joint bones are just floating in space… same with wrist->finger bones… you could use the wrists first child bone position to estimate the bones end point, but that might be the elbow to thumb, or elbow to pinky… which isn’t very accurate.
An alternative approach, could be to raycast against the skinned mesh… and find which bone the skinned mesh vertex is most highly weighted on, and assume that is the correct bone… at least that would be a sort of general solution and probably more accurate than trying to reconstruct bone start and end points.
You identify the bone by the weight of the nearest vertex acquired from your skinnedMesh raycast… then you have to figure out what bone that vertices largest bone weight index maps to, and .attach your decal to that bone.
What I want is to add a plane to the raycaster with the mouse and animate with the model, this works fine if the animation is not using bones but with bones, if I add the plane in the found mesh bassed on the raycaster intersection it dose not move with the model.
If a model had bones the only way to animate a plane is by being part of a bone.
Thank you I will look into it but whe I add the plane the bones are not animated, I just want to find the correct bone based on the mouse position on the model, it seems simple but is not
Guys, please open my eyes on this one I am literally lost, and it is the last pice of the puzzle to get my project finished…
I tried to use the example from Example : Raycast to Skinned Mesh but the class has some errors in it and I don’t know how to fix it…
I tried this but is not accurate far from it… if it would be possible to create a box exactly the same length as the bone around it this could work but from what I tried is not possible.
Is there really no way to get the most relevant bone based on the raycaster intersection, it seams like an impossible task…
It’s not impossible… but its not trivial. A bone is just a name for a rotation/scale/position. There’s no other information about it. No width/height/length. No shape… no mass. It has no dimensions. The only way you can get more information about it, is by inspecting what is bound to the bone, or seeing if it has children, and maybe making an educated guess about it… like… Oh this forearm bone has a position of 0,1,0… meaning it’s 1 unit away from its parent, the bicep, along the Y axis… so the bicep bone is probably 1 unit “long” but we don’t know how thick is is so maybe we guess that a bone is on average like 1/3d as wide as its length. This sorta works for big bones… arms/legs etc, but gets weird for bones in the hand/head/neck, etc. but maybe you can pretend small bones like that don’t matter. This process leads to something that looks like this:
A bunch of clickable bone “shapes” that you can raycast against… and it basically has to be hand-crafted per skeleton. Of course then if you place a plane, it will be placed on your hit colliders and not the actual skin… which isn’t what you want.
You can somewhat deeper information about bones inside the modeller, like its “envelope” size, but none of that info gets exported. Only position, rotation, scale, parent, children, gets exported.
Here’s what that bone envelope data looks like:
Still not super useful, but better than nothing(?)
You could write a script in blender python to analyse the skeleton and export more data about it, and then load that as JSON.
This still doesn’t solve your problem of placing a plane on the skin.
however:
If you invert your thinking in terms of casting against the skin… you can get a little more information about your guess. First off you will get pixel correct hit testing on the mesh… which is an improvement over the box-around-the-bone technique, but your guess of which bone was clicked, is still somewhat of a guess, since skin vertices can be bound to multiple bones. But the largest bone weight, is still a pretty good indicator that that bone is “the bone”.
Actual skinnedmesh raycasting isn’t trivial in and of itself, but has been addressed in threejs so you have a good shot of it working out of the box as long as your character has a proper bounding box and everything aligns just right.
Once you get the click, you have to look up the weights on that vertex… (i don’t remember how to do this… so you might have to research or ask chatgpt.) Then you’ll have to convert the weights index to the proper bone index… I think the bones are stored in a flat array somewhere in the skeleton… so that can get you the bone from the vertex weight+bone index.
Another option might be to raycast against something like the SkeletonHelper… that has a bunch of line segments, and you can set up the raycaster to treat lines as “thick” lines so that they have some dimensions to raycast against… that’s configured here: https://threejs.org/docs/#api/en/core/Raycaster.params
I.e. set the threshold in the lines to something larger than 1, but then you may still have trouble figuring out which bone is associated with that line segment, if its not just the parent… (which it probably is).
I know this is a an info-dump but… i’m just trying to impart to you some of what I’ve learned about skinned mesh animation over the last couple decades.
I actually tried chatgpt for this but the result is not finding the correct bone and I don’t have the revise the code and fix it because visually I don’t understand the concept of what is going on with these bones and meshes and the math relationship between them, this function has to work no matter what model I use since I am building a model viewer or at least I am trying too but this is the only thing that I can’t manage to accomplish and my skills are not there yet, all this stuff is new to me and is confusing …
Probably I will post this as a job and pay for it I don’t see any other way around this, it is giving me depression at this point, I understand now why most threejs beginners give up on this and I am an experienced developer and find all this stuff impossible at this point in my threejs journey, the main issue is that I can’t visualize what is happening and I am working in the dark…, I don’t want to think about what is happening in the brain of someone who is learning code as well
If someone from the community is willing to help me write this for me I am sure others will benefit it would be great if not is ok, I will post it as a job soon and I will post the answer for free here.
Thank you, guys, for the help, this community is awesome!
This is what I came up with using chatgpt, if you click on the model the plane will be added in the most weighted bone but is not accurate sometimes is added in the wrong bone and the plane is animated incorrectly you can try as well here Start
findClosestBone(raycaster, skinnedMesh) {
// Perform raycasting
const intersects = raycaster.intersectObject(skinnedMesh);
if (intersects.length > 0) {
// Get the intersected point in world coordinates
const intersectionPoint = intersects[0].point;
// Find the closest vertex to the intersection point
const geometry = skinnedMesh.geometry;
const vertices = geometry.attributes.position.array;
let closestVertexIndex = -1;
let closestDistance = Infinity;
for (let i = 0; i < vertices.length; i += 3) {
const vertex = new THREE.Vector3(vertices[i], vertices[i + 1], vertices[i + 2]);
const distance = intersectionPoint.distanceTo(vertex);
if (distance < closestDistance) {
closestDistance = distance;
closestVertexIndex = i / 3;
}
}
// Get the bone weights for the closest vertex
const boneWeights = geometry.attributes.skinWeight.array.slice(
closestVertexIndex * 4,
closestVertexIndex * 4 + 4
);
// Find the bone with the highest weight
let maxWeight = 0;
let maxWeightBoneIndex = -1;
for (let i = 0; i < 4; i++) {
if (boneWeights[i] > maxWeight) {
maxWeight = boneWeights[i];
maxWeightBoneIndex = geometry.attributes.skinIndex.array[
closestVertexIndex * 4 + i
];
}
}
// Access the skeleton to get the bone from the index
const skeleton = skinnedMesh.skeleton;
const maxWeightBone = skeleton.bones[maxWeightBoneIndex];
return maxWeightBone;
}
// No intersection found
return null;
}
I wrote what I was trying to describe to you… and it seems to work…
It checks against all skinnedMeshes in the scene, and shows the bone name in the gui.
look in script.js:
edit: I added an animation to it, and that seems to throw things off… not sure whats the deal… fun stuff
Yeah this stuff is really difficult it seams like an easy task but far from it.
The issue is that if the bone is not the right one the animation is messing up the plane position … so it has to be be the exact bone for the job when it animates,without the animation it looks ok…
Can I run the code that you wrote using the [Glitch Code Editor] online ?
This code is correct and provides a right bone index result based on the clicked, barycentrically-interpolated point. If it’s not working for you then it’s likely an issue in the way you’re using it. You need to provide small, minimal examples showing your work if you’re going to claim something doesn’t function otherwise no one can help you.
Ok guys so this is how is done, I am sure this will help others as well especially beginners.
// Create your raycaster and use the below code when clicking on the model.
let mousePosition....
raycaster.setFromCamera(mousePosition, camera...);
const intersects = raycaster.intersectObject(model..., true)
let intersection;
let meshName, mesh, boneName, bone;
let vertexIndex;
if(intersects.length > 0){
intersection = intersects[0];
if(intersection.face){
intersectionPoint = intersection.point;
vertexIndex = intersection.face.a;
if (intersection.object.isMesh) {
meshName = intersection.object.name;
mesh = intersection.object;
}
}
let bone;
if(mesh && vertexIndex){
let vi = vertexIndex;
let sw = skinnedMesh.geometry.attributes.skinWeight;
let bi = skinnedMesh.geometry.attributes.skinIndex;
let bindices = [bi.getX(vi), bi.getY(vi), bi.getZ(vi), bi.getW(vi)];
let weights = [sw.getX(vi), sw.getY(vi), sw.getZ(vi), sw.getW(vi)];
let w = weights[0];
let bestI = bindices[0];
for (let i = 1; i < 4; i++) {
if (weights[i] > w) {
w = weights[i];
bestI = bindices[i];
}
}
bone = skinnedMesh.skeleton.bones[bestI];
boneName = bone.name;
};
}
This can be written better I had to take it from multiple classes and create this hybrid but the idea is that this code takes the correct bone based on the skin weights and the intersection point.
