@Fennec First of all, all my thanks for your help, I couldn’t have pulled it off without you. One million thanks for putting the time for it, I hope this also helps and benefits you somehow (and all the community
).
Before all this I was rendering all the labels I needed in a secondary scene, computing the necessary un-projections/projections and it worked fine, but it was bottle-necking the CPU like crazy as I had to sync everything every frame, and when I had several thousand objects with labels I was hitting a hard ceiling performance wise.
But now that we made it work the problem is forever gone. In my specific use case it was quite hard as I don’t have just InstancedMeshes
but also BatchedMeshes
and even some special custom InstancedText
meshes extended from the super cool troika-three-text library.
To explain it a bit, my labels are made of:
- A shape that ends looking as the outline, they are a rounded rect so each one is a different geometry in a
BatchedMesh
.
- A shape over the previous one that looks as the actual background of the label, also a a rounded rect so also a geometry in a
BatchedMesh
.
- An optional Icon that has a ‘local’ offset (local relative to the rendered label) to fit in the right place with the text of the label. These are an
InstancedMesh
.
- The text of the label, using troika-three-text. These are most similar to an
InstancedMesh
.
Theory is fine, but this is the result (sorry for potato quality but the video has to fit 8Mb):
And as I always try to do in every thread I open, here is the complete working solution.
All the next code is meant to work with the awesome three-extended-material, which is the solution I use to be able to chain several custom shader modifications.
This is my final working extended shader code that I use in my InstancedMeshes:
import { GridThree } from '@/three/GridThree';
export const ExtensionLabelVertex = {
name: "extension-label-vertex",
uniforms: {
billboardMatrix: GridThree.billboardMatrix,
cameraZoom: GridThree.cameraZoom
},
vertexShader: (shader: any) => {
shader = `
uniform mat4 billboardMatrix;
uniform float cameraZoom;
attribute vec2 instLocalOffset;
${
shader.replace(
`#include <begin_vertex>`,
`
#include <begin_vertex>
vec3 toCamera = normalize((billboardMatrix * vec4(0, 0, 1, 0)).xyz);
vec3 up = (billboardMatrix * vec4(0, 1, 0, 0)).xyz;
vec3 right = normalize(cross(up, toCamera));
vec3 forward = cross(right, up);
mat3 billboardRotation = mat3(right, up, forward);
float scale = 1.0 / cameraZoom;
// Apply the offset before transforming the position
transformed.x += instLocalOffset.x;
transformed.y -= instLocalOffset.y;
transformed = billboardRotation * transformed * scale;
`
)
}
`;
return shader;
},
};
And this is the extended shader code I use for my BatchedMeshes:
import * as THREE from 'three';
import { GridThree } from '@/three/GridThree';
export const ExtensionLabelVertexBatched = {
name: "extension-label-vertex-batched",
uniforms: {
localOffsetsTexture: new THREE.Texture(),
billboardMatrix: GridThree.billboardMatrix,
cameraZoom: GridThree.cameraZoom
},
vertexShader: (shader: any) => {
shader = `
uniform sampler2D localOffsetsTexture; // Define local offsets texture
uniform mat4 billboardMatrix;
uniform float cameraZoom;
vec2 getBatchingOffset( const in float i ) {
int size = textureSize( localOffsetsTexture, 0 ).x;
int j = int( i );
int x = j % size;
int y = j / size;
return texelFetch( localOffsetsTexture, ivec2( x, y ), 0 ).rg;
}
${
shader.replace(
`#include <begin_vertex>`,
`
#include <begin_vertex>
vec3 toCamera = normalize((billboardMatrix * vec4(0, 0, 1, 0)).xyz);
vec3 up = (billboardMatrix * vec4(0, 1, 0, 0)).xyz;
vec3 right = normalize(cross(up, toCamera));
vec3 forward = cross(right, up);
mat3 billboardRotation = mat3(right, up, forward);
float scale = 1.0 / cameraZoom;
vec2 localOffset = getBatchingOffset( getIndirectIndex( gl_DrawID ) );
// Apply the offset before transforming the position
transformed.x += localOffset.x;
transformed.y -= localOffset.y;
transformed = billboardRotation * transformed * scale;
`
)
}
`;
return shader;
},
};
Also last but not least, this is the modification I had to do to the troika-three-text library for it to work just the same way (I copy just a part as the whole class is massive, DM me if you need the full code).
vertexTransform: `
vec3 toCamera = normalize((billboardMatrix * vec4(0, 0, 1, 0)).xyz);
vec3 up = (billboardMatrix * vec4(0, 1, 0, 0)).xyz;
vec3 right = normalize(cross(up, toCamera));
vec3 forward = cross(right, up);
mat3 billboardRotation = mat3(right, up, forward);
float scale = 1.0 / cameraZoom;
vec2 localOffset = troikaGetBatchingLocalOffset(${memberIndexAttrName});
// Apply the offset before transforming the position
position.x += localOffset.x;
position.y -= localOffset.y;
position = billboardRotation * position * scale;
mat4 matrix = troikaGetBatchingMatrix(${memberIndexAttrName});
position.xyz = (matrix * vec4(position, 1.0)).xyz;
${colorVaryingName} = troikaGetBatchingColor(${memberIndexAttrName});
${opacityVaryingName} = troikaGetBatchingOpacity(${memberIndexAttrName});
`
@Fennec As you mentioned this could be a very good foundation to incorporate this ‘billboard’ material into the engine, it works perfectly with Instanced and Batches meshes, even with third party libraries. SpriteMaterial
was always awesome but outdated as it doesn’t support instancing/batching.
Right now I’m absolutely drained to write it and propose a PR, also I think I lack the skill to do it so it works in all situations with N levels of parent objects…
At least I hope this post helps the community, just as @Fennec did to me, man you are the
!!!