Say I have a 3d model imported into my scene along with a separately created skeleton template. The skeleton is not bound to the 3d geometry. Are there any JS libraries to help calculate the bone weights associated to each bone? Pretty much how auto-skinning works in a DCC tool like Blender? I want it to be programmatic, so no advice like “take it into blender, press the auto-weight button, then export it”.
Right now I have a simple algorithm that just applies the weight with whatever bone is closest to the vertex. Just using a simple distance formula. You can imagine the results aren’t that great. I see there are a number of papers out there like this one (https://www3.cs.stonybrook.edu/~qin/research/2017-tvc-automatic-skinning-weight-retargeting.pdf). Not sure if anyone has actually turned some of these ideas into JS libraries - at least something that is open source that is usable.
I could start seeing how Blender does it, or read other papers. Didn’t know if any libraries existed before I might dig into the details with how this works.
Alright. Maybe there isn’t much out there that people know about. I am currently going in the direction of bone envelopes to associate the bones with the vertices, and assign all the indices and weights to the buffer attributes from that approach.
It looks like there are other neat ways that might even be better that deal with surface calculations, but I will try to exhaust bone envelope route first before.
Thanks for posting that @makc3d . I didn’t realize someone did a PR trying to add a simple bone skinning functionality to the editor.
it does look like the approach in the PR does some odd things, but I imagine it was never intended to be a great solution…but more of a proof of concept on the process. Currently I am doing it this way.
Going through each bone and do the following:
cast some rays perpendicular to the bone to find out where the bone hits the mesh
Build out a bounding box from those intersection points to fit the mesh with a bit of extra padding (things call this a bone envelope)
See what vertices are inside this bounding box with an intersection test. These intersection points will get assigned to the bone
Once that process goes through each bone, I look to see any vertices that didn’t get associated with any bone. I just use a closest distance formula for that and assign that bone 100% of the weight.
I am currently refining how the bounding box (bone envelope) shape looks to be smarter. Some areas like limbs are fine, but other areas like the shoulder have too large of a bounding box. The shoulder one is casting down all the way to the feet, which is obviously going to be too large. I might start using bone parents and bone children to help constrain the size.
Raycasting doesn’t sound like a great way to do the distance calculations…
Parents and children sound like the way to go…
I think you want to loop through all vertices, for each vertex, compute distance to each bone line segment, then pick the nearest bone(s) with some thresholding.
This could probably get you usable results for some organic objects/tentacles/trees, but rigging animal or human characters rarely works well with automated solutions.
People write entire thesis papers on automated solutions and afaik, there aren’t many silver bullets. Mixamo does a pretty good job of auto rigging but human/animal forms can be pretty unforgiving… and it uses a whole machine learning model to do it.
It’s definitely not a case of “this is so easy why hasn’t someone done it before”…
But I’d love to find out otherwise! keep us posted!
I am doing raycasting to help generate an “envelope mesh” for a bone. This envelope mesh is pretty much just a bounding box along the length of the bone. I can make it grow outward and will help determine what vertices belong to the bone based off the intersection. I will see how good, or bad, the results will be doing it this way. I might play around with a couple different approaches and see what works between closest distance, bone envelope, and another one with voxelization that I read about.
It does seem like there is a bit of art with how these solutions are composed, given there are so many variations on types of characters or objects that have skeletons. I can see why solutions like Mixamo kind of stopped with humanoids and didn’t go into other areas like quadrupeds like dogs, cats, and dragons. I think Mixamo even has a custom solver just for hands to determine how to do the weights for that area.
If I come up with anything useful I can share it. I have some future ideas as well with this, but don’t want to get ahead of myself.
Link on the bottom left goes to the github page. I have a few different formulas for the skinning process. The bone envelope is the most complex, but still has issues with certain situations that I need to account for. I will try to keep improving this as a side project for fun.
