What are the strategies for achieving this?
I was thinking maybe I can modify the skinned shader chunks and replace some of those calculations with instanced buffer attributes. I’m not sure how the bone texture and all that apply and where I would make the change.
The other option I was brainstorming would be to have one animate skinned mesh as the “master copy” and then several instanced meshes with a custom shader + some texture magic (similar to bone texture) for positioning based on frames after having calculated every frame on the cpu from the master copy once. This is possible because 1) there max 7-10 frames (time slots) and 2) they are 1 mesh, low poly ~2000 vertices per type of mesh and 3) I don’t even need interpolation between frames since I’m retrofitting a legacy game.
It depends on the amount of these meshes / instances - but doing keyframe calculations on the CPU may be less performant than just not doing instancing altogether (without instancing you’ll also be able to optimise animations with LOD levels and updating only the objects close enough to the camera.)
If nevertheless your scene is indeed packed with billions and billions of these meshes - this fork may be just what you’re looking for. It’s based on this discussion - wasn’t merged to the core though, you’ll need to add it manually.
There’s also this - but it’s WebGPU only and not super widely supported at the moment.
I would do all the calculations once up front and bake them into a texture. Eg 20 frames x Vector3 = 90 Height X 2000 Vertices Width
I might try that branch out. A typical maxed out scene would be ~400 units per player, at < 2000 triangles a unit. Up to 8 players. In addition to the terrain (50k triangles divided by 8x8 geometries for culling/LOD) and other about 150-200 small items on the map (“non player units”). The camera is typically looking at about 1/8 to 1/4 of the entire map. In total there are about ~100 unit types, so 100 unique textures / meshes + 64 textures/meshes for the terrain. This is all discounting particles/overlays which is another thing altogether.
I need to round out my understanding of what I need to be concerned about when using instancing.
b) I guess I cannot use LOD with instancing?
c) How do I efficiently omit drawing without relying on “count” as I don’t know if I can use that without reshuffling everything.
There is a difference between instances being individually animated and all instances sharing the same animation state which would come by default if not extended to an texture atlas.
You can more or less easily expand the current system for instancing, essentially you will need skeletons sharing the matrices buffer and all be packed into it, writing into it with their offset they been assigned to as well as the shader reading from it. Essentially a texture atlas for the bone texture.
Since you need a animation mixer for animation it would be more of a entirely additional task to move the animation processing onto GPU, i just throttle and reduce animations based on distance and other, it should totally suffice. ( you can also do by mixing 2 frames and linear lerp between their matrices, a bit more work but will still animated smooth with little detail loss)
You don’t have LOD regarding geometry unless you use one instance stack per LOD.
You can limit the amount of instances dynamically, just the buffer should be fixed. You could use different techniques to expand that or chunk it if needed.
Well no, but yesterday evening i gave it a more deep thought and i will make a plugin for it soon with a simple to use API so your actual mesh object can stay in the scene as usual.
The point is that LOD can become a must, less regarding geometry, more regarding the bone texture. The more bones per mesh the larger the bone texture atlas becomes, assuming your character is detailed with fingers and facial bone rig, going with round about 100 bones, 100 meshes would already take up a 400x400 32bit texture that is uploaded every frame.
So my idea is to LOD the bones as well just as i already skip their computation, you will set a priority on bones so the lower priority ones in a lower LOD will not be included, resulting in a smaller bone texture - essentially only closer characters will require the highest LOD where the texture can be updated the partially to the used amount.
It should be enough for most causes to throttle and reduce animations by LOD, to max this out animations could be threaded, but that’s a bit more complex and more work and the question is if it’s necessary, in the end the main constraint remains the upload to GPU of the transformations. But depending on the asset/bone count it should be possible to get several hundreds to several thousand for a real world game with the LOD system, but using a spatial index obviously would give it more juice as for a regular scene the plugin will just as the renderer process things linear and without culling.
Implemented it so far, instanced skinned mesh with individual animation states, also with the LOD system in-built to not only use a reduced geometry but also reduced set of bones.
The strategy is that instances are usually not all in front of the camera, so most likely fall into the lower LOD depending on the distance you set for them, generally it’s about keeping uploads of bone matrices small as possible asides of reducing animation computations.
Still need to test through the LOD part further and see how far i can reduce animations without core modifications, but they could be already throttled by the LOD using the onUpdate callback to update your animation mixer.
The general idea is that each instance only uses 1 Mixer for each LOD, so the reduced bones are remapping the skin weights to parents for those bones not used in the LOD. Otherwise you would obviously need 3 Mixer for 3 LODs.
The other benefit is you can attach things and items as you would usually regardless of the LOD. And you can unlink it from the instancer anytime so it will render as regular SkinnedMesh again.
So far the API looks like (for 1 LOD)
const mixers = new WeakMap;
// Creates a instancer for this type of asset, you don't use the original of the glTF in the scene but make clones from it
instancer = new SkinnedInstancer({
target: playerMesh,
maxCount: 1024,
levels: [
{
distance: Infinity
}
],
onUpdate: ( instance, delta ) => mixers.get( instance ).update( delta )
});
scene.add( instancer );
// Clones the mesh and it's skeleton appropriately and
// calls instancer.link( mesh ) to render it through the instancer, but you
// still add and use the clone as usually in the scene.
const player = instancer.cloneLink( playerMesh );
mixer = new THREE.AnimationMixer( player );
mixers.set( player, mixer );
scene.add( player );
That’s mighty impressive. I hope to get my hands on that!
After playing around a bit I’m realizing my original implementation was assuming I could have an instanced attribute per vertex but I’m finding out I cannot! At most I can have a matrix4 (instanced buffer attribute) per instance, which definitely ups the challenge. One of my challenges is to keep the interface as uniform as possible because I’m not only supporting 3d models but the classic 2d graphics (and in fact they can change depending on active camera and can also coexist!). So I have an image pool, and it’s responsible for handing out 3d objects, which now will be more like proxies (instances) even if they are actual individual meshes in some cases. I think that would tie in nicely to your API.
I have looked partially into spatial indexing but have not needed it yet. I see the benefit but at a non-trivial complexity cost!
That LOD system sounds pretty nuts. I would imagine that is quite complex algo-magic and above my head. I was originally just planning on doing 2 LOD levels manually and hadn’t considered yet what to do with animations. The only other thing I can think of is the integration with three materials since I do some team colors modifications, emissive intensity and such.
So automatic bone reduction works now, but the mesh should not have errors like the Soldier.glb in the three examples, it has skin indices pointing to bones it does not has.
Anyway with any other models i tested there seems to be no issue, the only thing left to do is figuring out why THREE on the render call still is doing some matrix computations, and unfortunately i don’t have a asset with multiple geometry LODs here to test i could also use for a demo, but there should be no issue as long as every LOD is assigned to the same skeleton.
It also skips processing animation bindings or matrix computations on inactive bones. With debug mode enabled it will visualize the LOD levels.
Max instances (per LOD) 256 (so 512 in total)
LOD 0 texture 32x32, bone count 67, texture 1 MB
LOD 1 texture 8x8 ,bone count 14, texture 0.0625 MB
are you planning to make this available to use in the near future?
is it possible to control the animation state individually / instance? It’s fine if it has to be the same animation, but would it at least be possible the control the current time and/or perhaps the speed?
@Fyrestar this is amazing, will this be an external jsm “addon” module or are you planning to make a pr on the main three.js library?! Exciting stuff! Can’t wait to see it in action!!
