Generate InstancedMesh from dynamically imported file

Hi,

Short version, my project aim to have a converter of IFC files to glTF files. This is already done, but now I want to optimize things so that I can load really huge IFC files. So I use InstancedMesh but the rotations are messy.


Longer version:

Context

Let’s say I have a file picker that allow the user to import a 3D model file (.glTF, .fbx, whatever). This file may have several objects which are the same mesh. For instance, we could have a house with the same window object several times.

To optimize the performances, I want to detect the objects that could be instanced and then convert the imported file to a new glTF file, with instancedMesh in it (using the EXT_mesh_gpu_instancing extension ).

Algorithm

So I followed these steps:

Similar objects detection
In my case, I have metadata that allow me to know if it is the same mesh or not. But I guess you could compare the vertices to know if the meshes are identical

I loop over the meshes and store them in a data structure by “type”. So if several objects are similar I group them, if an object have no similarities with other objects it is stored alone.

Loop over every group of object
Now that we have different groups, I loop over them.

If the group contains only one element, there is no need to make an instancedMesh so I generate a mesh of it and add it to the scene node.

If the group have several objects I do the next steps

Get relative positions
I loop over the similar objects and get the position of each object. We have two cases:

  • The origin of the object is its center
  • The origin of the object is (0;0;0)

First case is easy, we already have the position of the future instances

Second case, we have to get the center of each object. Then we take the first object of the group and determine the position of other objects relatively to this first object.

Now we have the position of every objects

Get relative rotations
Now the tricky part and where I’m stuck.

Same as for the position, we need to get the rotation of every object relatively to the first object of the group.

My method was to loop over the first three vertices of each to get a triangle, compute its normal and get the angle between its normal and the one of the first object. So for the first object, the angle is 0 but others we have a certain angle. Since, we have similar objects, I assume that this triangle is the same for every objects.

const vertices = newGeometry.attributes.position.array;
let index = newGeometry.index.array[0];
const vec1 = new THREE.Vector3(...vertices.slice(3 * index, 3 * index + 3));
index = newGeometry.index.array[1];
const vec2 = new THREE.Vector3(...vertices.slice(3 * index, 3 * index + 3));
index = newGeometry.index.array[2];
const vec3 = new THREE.Vector3(...vertices.slice(3 * index, 3 * index + 3));

const xVector = new THREE.Vector3().subVectors(vec2, vec1);
const yVector = new THREE.Vector3().subVectors(vec3, vec1);
const normal = new THREE.Vector3().crossVectors(xVector, yVector).normalize();

From this angle, I create a rotationMatrix for every object and store it.

const rotationMatrix = new THREE.Matrix4();
rotationMatrix.lookAt(vec1, vec2, normal);

const quaternion = new THREE.Quaternion().setFromRotationMatrix(rotationMatrix);

Generate instances
The final step is to create the instancedMeshes. So here I loop over the groups of similar objects.

For each group, I generate and InstancedMesh with the number of similar objects of this group as length. Then I apply position, rotation and scale:

  • Position: I copy the position of the center of each object
  • Scale: (1;1;)

And then rotation…

Here I do this steps:

  • Get the rotation of the first object (quaternion)
  • Invert it (quaternion.invert)
  • Multiply it by the rotation of the object that we got at the previous step (quaternion.multiply)
const matrixRotation = new THREE.Matrix4();
matrixRotation.identity();

const quaternion = new THREE.Quaternion();
quaternion
	.copy(firstobject.quaternion)
	.invert()
	.multiply(object.quaternion);

Finaly, I compose a matrix:

matrix.compose(position, quaternion, scale)

Basically, it works for some items but not the entire scene.

Here are the IFC file that I’m testing for my project. It’s an IFC test file used by the library IFC.js. I also made an glTF of it so you can see what it looks like but is not with instancedMesh.

sourceFile.glb (331.2 KB)
sourceFile.ifc (693.5 KB)

Here is the result I get with the algorithm I described (the source file is colorless on the screen but it has colors):

As you can see, the chairs don’t have the right rotation, and on the outside, there are some bars around the window (on the left) that do not have the right scale (I don’t know if it is a scale issue or rotation one).

Here is the file of these screenshots:
scene.glb (321.6 KB)

Do you have any tips on how to have the right rotation?

A shortcut instead of hand computing the quaternion… you can use firstObject.attach( secondObject );

The .attach method is different than .add method. It keeps the visual transform of the second object, while changing its parent to the first object.

After the .attach, then secondObject.quaternion should be the quaternion you need (maybe if I’m understanding) ?

https://threejs.org/docs/#api/en/core/Object3D.attach

1 Like

Could be possible. But (I forgot to precise it), if the objects have their position at the center of the scene (0;0;0), we can imagine that their rotation is also (0;0;0;1). So with your method I can’t get any rotation.

To sum up, one of the edge cases is when the 3D model is defined by the vertices. In that case, the position and rotation of the object is the origin.

So, I was looking for a solution on maths forums and I found this:

I tried it. So basically we have the equation P=RQ where:

  • P is a 3x3 matrix with 3 vectors of an object you want to get the rotation of
  • Q is a 3x3 matrix of the first object of the group

R=PQ^−1

Here we have:

const matP = new THREE.Matrix3(
	object.xVector.x,
	object.yVector.x,
	object.normal.x,
	object.xVector.y,
	object.yVector.y,
	object.normal.y,
	object.xVector.z,
	object.yVector.z,
	object.normal.z,
);
const matQ = new THREE.Matrix3(
	firstobject.xVector.x,
	firstobject.yVector.x,
	firstobject.normal.x,
	firstobject.xVector.y,
	firstobject.yVector.y,
	firstobject.normal.y,
	firstobject.xVector.z,
	firstobject.yVector.z,
	firstobject.normal.z,
);

let rot = new THREE.Matrix3();
rot = matQ.invert().premultiply(matP);

So now I have this:

So we have the right rotations!


The last bug is with the windows. But I think I know why it’s bad. I think, I labelize the window frame (composed of 4 objects) as the same 4 objects, while they don’t have the same size. Horizontal ones are large, while vertical one are small.

I thought I solved the issue of scaling by getting this ratio:

const scale = new THREE.Vector3();
scale.x = objectBoundingBoxSize.x / firstobjectBoundingBoxSize.x;
scale.y = objectBoundingBoxSize.y / firstobjectBoundingBoxSize.y;
scale.z = objectBoundingBoxSize.z / firstobjectBoundingBoxSize.z;

But I had some issues with the windows. As you can see there are gaps:

So I modified my " Similar objects detection" algorithm to separate object of different sizes, even if the geometry is the same. It is less effective but it does the work for now. So if anyone has an idea, please share

this tool does it automatically GitHub - pmndrs/gltfjsx: 🎮 Turns GLTFs into JSX components

open console, type:

# only instance re-occuring parts
npx gltfjsx yourfile.glb --instance

# instance all meshes, the part can be re-used in threejs and will be auto-instanced
npx gltfjsx yourfile.glb --instanceall

here are two videos showing how it works:

https://twitter.com/0xca0a/status/1624061030354546695

https://twitter.com/0xca0a/status/1552372534397501446

basically in a declarative world the model structure is known compile time. the same structure can now be expressed with drei/Merged & drei/Instance which imo is critical since you need to retain the natural nesting of the part. it doesn’t require anything else, the meshes don’t have to be instanced in blender. this is possible with a gltf-functions transform, i think it was either dedup or join, it looks for meshes with the same or similar vertices and removes duplicates, which makes it easy to find candidates for instancing.

i have not seen anything like this for vanilla, i don’t believe it is possible — other than messing with specific code for specific models. a three.instancedMesh is a glob, it has no nesting, this is the first hinderance. if you had something that allowed instances to be similar like THREE.Object3D then it would be possible, i would start there.

thinking of it, GitHub - donmccurdy/glTF-Transform: glTF 2.0 SDK for JavaScript and TypeScript, on Web and Node.js. has a in-model instancing transform. it dedupes your model and instances similar models. you wouldn’t be able to re-use the model multiple times and retain the number of instances and the parts are probably static from now on, so you can’t move, for instance, a chair around. but if that’s acceptable the tool will probably work for you.

Yep I’m using Vanilla but thanks anyway

Yes I investigated it but I was already deep in my code so decided to finish it anyway