"lookat" / "billboard" vertex shader for InstancedMesh instances?

Questions
, , , , , ,
1

I have a large InstancedMesh object with many simple geometry instances. I would like to write a vertex shader in TSL that makes each instance always face the camera, like billboard sprites do.

The following TSL node works well for making a regular Mesh object always face the camera:

material.positionNode = material.positionNode = tsl.Fn(() => {
	const objectCenter = tsl.modelWorldMatrix.mul(tsl.vec4(0.0, 0.0, 0.0, 1.0)).xyz;
	const up = tsl.vec3(0, 1, 0).toVar();
	const toCamera = tsl.cameraPosition.sub(objectCenter).toVar();
	// set toCamera.y = 0 to only allow rotation around the y-axis (i.e. make it "cylindrical")
	toCamera.assign(tsl.vec3(toCamera.x, 0, toCamera.z).normalize());
	const right = up.cross(toCamera).normalize();
	up.assign(toCamera.cross(right).normalize());
	const rotationMatrix = tsl.mat3(right, up, toCamera);
	return rotationMatrix.mul(tsl.positionGeometry);
})();

However, using this TSL node on an InstancedMesh doesn’t do anything (the instances don’t face the camera like they should).

What do I need to change in order for this to work for individual instances in an InstancedMesh?

2

In general, instance’s center is: modelWorldMatrix * instanceMatrix * vec4(0, 0, 0, 1) :thinking:

3

Here is something interesting though: as long as the camera is at (0,0,0), then the instances rotate properly to face the camera. But as soon as I travel away from (0,0,0), the instances stop facing the camera and go back to their “default” rotations.

4

Is there something stopping you from using the setMatrixAt method using a dummy object to lookAt the camera first?

5

There are hundreds of thousands of instances in my InstancedMesh, I wouldn’t want to loop through each one of them in JavaScript on every frame.

6

modelWorldMatrix gives the same transform for all instances in your InstancedMesh, that’s why it only works when camera is at (0,0,0)

use a storage buffer to access each instance’s individual matrix

import { storage, StorageInstancedBufferAttribute, instanceIndex } from 'three/tsl';

function setupInstancedBillboard(instancedMesh: InstancedMesh) {
    const material = new MeshBasicMaterial();
    
    // Storage buffer to access per-instance transformation matrices
    const instanceMatrixStorage = storage(
        new StorageInstancedBufferAttribute(instancedMesh.instanceMatrix.array, 16)
    );
    
    material.positionNode = tsl.Fn(() => {
        // Get each instance's individual world center
        const instanceMatrix = instanceMatrixStorage.element(instanceIndex);
        const objectCenter = instanceMatrix.mul(tsl.vec4(0.0, 0.0, 0.0, 1.0)).xyz;
        
        // Rest is the same as your original code
        const up = tsl.vec3(0, 1, 0).toVar();
        const toCamera = tsl.cameraPosition.sub(objectCenter).toVar();
        
        // cylindrical billboard (Y-axis rotation only)
        toCamera.assign(tsl.vec3(toCamera.x, 0, toCamera.z).normalize());
        
        const right = up.cross(toCamera).normalize();
        up.assign(toCamera.cross(right).normalize());
        const rotationMatrix = tsl.mat3(right, up, toCamera);
        
        return rotationMatrix.mul(tsl.positionGeometry);
    })();
    
    return material;
}

I used this approach in my webgpu octahedral impostor forest demo (which also uses always camera facing quads and worked fine with more than a million such instances (on my 3090 though ^^))

7

post removed–see solution in my next post

8

This ended up being the solution for me. Visit this link for more info on the following code snippet: TSL: `instanceMatrix` not available in `positionNode` for use in `InstancedMesh` · Issue #31659 · mrdoob/three.js · GitHub

material.positionNode = tsl.Fn(({object: mesh}) => {
    const objectCenter = getMatrix().element(3).xyz;
    const toCamera = tsl.cameraPosition.sub(objectCenter).toVar();
    // set toCamera.y = 0 to only allow rotation around the y-axis (i.e. make it "cylindrical")
    toCamera.assign(tsl.vec3(toCamera.x, 0, toCamera.z).normalize());
    const up = tsl.vec3(0, 1, 0).toVar();
    const right = up.cross(toCamera).normalize();
    up.assign(toCamera.cross(right).normalize());
    const rotationMatrix = tsl.mat3(right, up, toCamera);
    return rotationMatrix.mul(tsl.positionGeometry);

    function getMatrix() {
        if (mesh.isInstancedMesh) {
            // Can I use tsl.instance() to make this code cleaner?
            // I tried using tsl.instance().instanceMatrixNode but it's always null.
            // Leaving this line here but commented out.
            // tsl.instance(mesh.count, mesh.instanceMatrix).toStack();
            const attribute = mesh.instanceMatrix;
            const matrices = attribute.array;
            if (mesh.count <= 1000) {
                const bufferNode = tsl.buffer(matrices, 'mat4', Math.max(mesh.count, 1));
                return bufferNode.element(tsl.instanceIndex);
            } else {
                const buffer = new three.InstancedInterleavedBuffer(matrices, 16, 1);
                let bufferFn = tsl.instancedBufferAttribute;
                if (attribute.usage === three.DynamicDrawUsage) {
                    bufferFn = tsl.instancedDynamicBufferAttribute;
                }
                // F.Signature -> bufferAttribute( array, type, stride, offset )
                const b0 = bufferFn(buffer, 'vec4', 16, 0);
                const b1 = bufferFn(buffer, 'vec4', 16, 4);
                const b2 = bufferFn(buffer, 'vec4', 16, 8);
                const b3 = bufferFn(buffer, 'vec4', 16, 12);
                return tsl.mat4(b0, b1, b2, b3);
            }
        }
        return tsl.modelWorldMatrix;
    }
})();