InstancedMesh2

agargaro · November 27, 2023, 3:11pm

Hi, I have been working on an extension to InstancedMesh for a couple of days now.

I have created my own InstancedMesh2 class that extends InstancedMesh, which manages an array of InstancedEntity, itself a class that is similar to Object3D that allows you to manage the position, scale, rotation and visibility.
Visibility is managed by swapping the element you wish to hide with the last one in instanceMatrix array, and decreasing the count by 1.
This allows the hidden element not to actually be rendered.

This approach uses more memory but:

it simplifies object transformations
it allows hiding all instances that are not in the camera frustum (I am working on this with good results, using a custom octree and other optimisations, even if i want to try BVH)
it allows the easy creation of an InstancedLOD (which I am working on, also with good results)

I tried using an InstancedMesh2 with a count of 2kk, using low poly geometry.
In a favourable scenario, with static objects and camera, thanks to the frustum control, it was possible to hide 85% of the instances. Frustum check needs to be done after each camera transformation (still considering static objects).

How to use it

const monkeys = new InstancedMesh2(geometry, material, count, {
  behaviour: CullingDynamic,
  onInstanceCreation: (obj, index) => {
    obj.position.random();
    obj.scale.setScalar(2);
    obj.quaternion.random();
  }
});

scene.add(monkeys);

// How to handle instances
monkeys.instances[0].visible = false;
monkeys.instances[1].rotateOnWorldAxis(xAxis, Math.PI);
monkeys.instances[1].updateMatrix();
monkeys.instances[2].position.x += 10;
monkeys.instances[2].scale.multiplyScalar(2);
monkeys.instances[2].updateMatrix();

Live Example (no frustum check)

I will make all my work public and usable by anyone, as packages with only three.js as peer dependency, and would love to receive advice.

I thank @DolphinIQ for the advice

Currently a similar version without visibility management, but with interaction events is already available in my three.ez library.

In future posts I will explain how I plan to optimise visibility management according to the frustum of the camera and perhaps how I would like to manage InstancedLOD.

Update:

Last example (1 million instances + frustum culling)

Package released

PavelBoytchev · November 27, 2023, 11:13pm

Would this change the order of meshes?

Apart from the matrices, other instance data should also be changed (e.g. colors and any custom attributes).
In some cases I have instanced mesh accompanied by a parallel array of additional non-graphical instance data (mapped 1-to-1 to the instances data). Exchanging instances would break this mapping.

agargaro · November 28, 2023, 12:11am

Great observations, as usual

No, the instances array will remain the same.
There is an internal, invisible to the user, handling of the instanceMatrix and instanceColor indexes.

instancedMesh.instances[0].visible = false;
// instancedMesh.instances[0] is still the same element.

Both matrices and colors (if present) are currently swapped.
Custom attributes may be swapped with a utility.

This is even simpler.
You don’t need an array, but to add the data to the instance.

class Entity extends InstancedEntity {
  customData = 10;

  constructor(parent: InstancedMesh2, index: number) {
    super(parent, index);
    // set position, scale, quaternion.
  }
}

const myInstancedMesh = new InstancedMesh2(geometry, material, 1000, Entity);
myInstancedMesh.instances[0].customData = 15;

Do you think something can be improved?

PavelBoytchev · November 28, 2023, 9:56am

Hm. Does it mean that matrices and colors are swapped internally, but the user must swap all other THREE.InstancedBufferAttribute arrays?

Does it have cloning? (De-)Serialization?

Any plans to make BatchedMesh2 over the expected BatchedMesh?

agargaro · November 28, 2023, 2:24pm

You are right. It should be possible to swap also InstancedBufferAttributes without the user doing it manually.

However, geometry can no longer be shared (only if has custom attributes) and InstancedBufferAttributes with meshPerAttribute greater than 1 would not be compatible.

Right, they are methods that must be there, but I will include them at the end.

I’m waiting for r159, but if you think it would be useful, I’ll gladly do it.

github.com/mrdoob/three.js

BatchedMesh: Provide approach for sharing geometry and transforms between BatchedMesh

opened 11:51AM - 17 Nov 23 UTC

gkjohnson

Enhancement

### Description Right now you can render a number of geometries with a single m…aterial using a BatchedMesh but you may want to swap materials of an individual mesh for things like selection, highlighting, etc. Right now that means creating and maintaining a completely different BatchedMesh which is slower and takes more memory. Instead it would be best if two BatchedMeshes could share geometry and potentially transform data so they can both specify different materials and visibility arrays. ### Solution I'm not exactly sure of some of the options here - but one solution is to add a "BatchedBufferGeometry" object that BatchedMesh references and stores all the geometry, draw ranges, etc. The BatchedMesh would still bookkeep the multidraw buffers, etc: ```js const mesh1 = new BatchedMesh( maxGeometryCount, maxVertexCount, maxIndexCount, material1 ); const index = mesh1.geometry.addGeometry( ... ); mesh1.setMatrixAt( ... ); const mesh2 = new BatchedMesh( mesh1.geometry, material2 ); mesh1.setVisibilityAt( 0, false ); mesh1.setVisibilityAt( 1, true ); scene.add( mesh1, mesh2 ); ``` ### Alternatives Instead the end user could just use something like a proxy or wrapper class that forwards all common information between the BatchedMeshes while reimplementing all needed: ```js const mesh1 = new BatchedMesh( ... ); const mesh2 = new BatchedMeshWrapper( mesh1 ); ``` Or - possibly the better approach - BatchedMesh could support different and arbitrary material properties and textures but this can get complicated. It's possible this would be best done with node materials in the future. ### Additional context User mentioned this need in three-mesh-bvh: https://github.com/gkjohnson/three-mesh-bvh/issues/513

I saw this issue on BatchedMesh and I have the impression that we are doing similar work… I’d love to talk to him.
InstancedMesh2 already allows you to share entities with other InstancedMesh2 (useful for InstancedLOD, but needs improvements).

  // this is still working in progress, will be easier to use
  const lod = new InstancedLOD(count)
  lod.addLevel(new THREE.IcosahedronGeometry(1.5, 16), material, 0.5);
  lod.addLevel(new THREE.IcosahedronGeometry(1.5, 8), material, 3);
  lod.addLevel(new THREE.IcosahedronGeometry(1.5, 4), material, 10);
  lod.addLevel(new THREE.IcosahedronGeometry(1.5, 2), material, 20);
  lod.addLevel(new THREE.IcosahedronGeometry(1.5, 1), material, 50);
 
  for (let i = 0; i < lod.count; i++) {
    lod._sharedData[i].position.random().multiplyScalar(1000).subScalar(500);
    lod.updateInstancedMatrix(i);
  }

agargaro · November 28, 2023, 2:37pm

Small experiment with InstancedLOD and 2kk meshes.
This is also a favourable scenario, where 88% of the meshes were hidden thanks to the frustum control
The goal is to try to have similar results by calling up a continuous update of distance and visibility calculation, although I don’t think it will be possible, even if @gkjohnson made me believe in magic with three-mesh-bvh

Little memory problem:

agargaro · December 3, 2023, 10:12am

I have prepared an example where it automatically swaps InstancedBufferAttribute.
In addition, I added the possibility of passing a callback executed for all instances.

const monkeys = new InstancedMesh2(geometry, material, 20000, (obj, index) => {
    obj.position.x += index * 5 - 20;
    obj.scale.setScalar(2);
    obj.updateMatrix();
});

Thank you very much for the advice

agargaro · December 10, 2023, 5:08pm

Small update:

I wrote an algorithm to minimise the swaps made in the array to manage visibility, and implemented a first test of frustum culling.

Can you please tell me if this example (50k animated meshes) runs at 60 fps on your computers?

PavelBoytchev · December 10, 2023, 5:14pm

Oscillates in the range of 60-62 fps.

agargaro · December 10, 2023, 5:19pm

Thank you Do you think this is an acceptable result? (I don’t know your monitor refresh rate )

Currently a check is made on all meshes if they are inside the frustum (like BatchedMesh i think).
This is only useful if all meshes are animated every frame, otherwise there will be a flag indicating that the scene is static and in that case I can optimise a lot thanks to a hierarchical control.

And it will also be possible to optimise this example with an InstancedLOD ofc.

PavelBoytchev · December 10, 2023, 8:43pm

The result looks nice, however … I don’t know what is acceptable. For every person it means something different (my card can reach 144 fps for simple scenes, and 60 fps for your demo).

Such performance tests are good for developers, but are not indicative for the end user … I mean, I have had many cases, when something looks fast on a demo page, but performance is notably worse when applied on a real page (with other objects, various materials, animations, effects, etc).

Edit:

Here is something strange. When I fly towards the most distant Suzanne, the others go outside the frustum. I expect this should increase the fps if frustum culling is active. But even when I end up with just one Suzanne, the fps is barely increased by 10% to 66 fps.

agargaro · December 10, 2023, 8:53pm

Probably in this case the slowness is caused by animating 50k meshes every frame.

Here is an example without animation, this one I hope doesn’t go below 144…

PavelBoytchev · December 10, 2023, 9:00pm

Yes!

agargaro · December 10, 2023, 9:10pm

Well this looks like a good result, thank you very much for your help!

In a few days I hope to publish some examples of the InstancedLOD class.

Harold · December 10, 2023, 10:45pm

165 (my refresh rate) at 100.000 (animated) items on an RTX 2080ti.

agargaro · December 10, 2023, 11:14pm

Thanks for the test
I think you also have a high-end cpu to generate all those frames.
Updating 100k matrices for 165 frames is a heavy load for the cpu…

…

But it doesn’t really make sense to update the matrices of hidden objects… just like three.js does… I really need to improve this…

Harold · December 10, 2023, 11:22pm

Oh right, the CPU may have a role to play here. I’m running on an Intel I7 13700K.

manthrax · December 11, 2023, 6:20am

solid 144 fps on my gtx1080 when camera is in the center, and ~120 when zoomed out…

agargaro · December 14, 2023, 12:00am

Small update:

I optimised the update of the matrices… now only the visible instances matrices are updated.

@PavelBoytchev Now you will probably make more fps even with animated instances

agargaro · February 13, 2024, 12:04am

After a break, I resumed work on this class.

I implemented optimisations on static instancedMesh (without animations) using a bvh and Hierarchical Culling.
There are still things to improve but it is a good start.

Without a LOD the gpu works too hard, so that will be the next step.

Can someone test it and tell me the fps, to see if there is any improvement compared to the last example? Thank you.

Topic		Replies	Views
How to do frustum culling with instancedMesh? Questions geometry	8	5541	February 17, 2023
Invisible InstancedMesh Instances Questions instanced-mesh	2	1324	November 24, 2021
Instance mesh culling Questions instanced-mesh	3	1397	April 15, 2021
Debugging culling in InstancedMesh and Orthographic camera Questions camera , orthographic-camera , instanced-mesh	0	474	December 23, 2022
Mixed use of Mesh and InstancedMesh, R120 version reports an error Questions instanced-mesh	1	1288	August 30, 2020