Drawing many text labels using html <canvas> overlay

Questions
1

Hi,

I have an instancedMesh of sphereGeometries in react-three-fiber.
I would like to add labels to these sphereGeometries. I am aware that performance will be reduced significantly if I try to create many html labels. Therefore I would like to write the labels to an html <canvas> element that is overlaid over the react-three-fiber <Canvas> component.

To transform 3D coordinates to 2D screen coordinates I would need to use THREE.Vector3.project(camera). Hence, I would need to get access to the camera object using say const camera = useThree(state => state.camera).

useThree() is however only happy if it is called from within a child component of <Canvas>.

How would one go about creating an html <canvas> component that overlays the react-three-fiber <Canvas> component and draws 2D text projected from 3D coordinates?

2

I’d just use the Text component from Drei. Your solution is going to be complicated to code and the text will look ugly due to aliasing.

3

What do you mean by projected, and how are you planing on projecting it? … Just curious!

4

I want to second the use of drei Text ( i think its using Troika text under the hood )
The quality is waay higher than you’ll get with canvas, there’s no annoying management of multiple canvases/overlays/html reflow…

5

I’ve asked about the projection, because I was wondering if the OP meant literally projecting a canvas over the instanced meshes, which can be done with a Spotlight projection.

Here is another one projecting an image (It’s hard to get the alignment right, you have to do a lot tweaking).

drei Text or troika text is expensive, defeating the purpose of using instanced meshes in the first place, and it’s a mesh with its own material and geometry,it can’t be wrapped around a sphere.

Since we’re talking about troika, it just so happened that the same library offers an extended version of InstancedMesh, instancedUniformsMesh, where you can set custom uniforms and update them for each instance, with a custom shader you can set a map uniform and individually update its value for each instance. (check the examples at the bottom page)

6

Hi,
I will give drei Text a try.
Wrapping text over the instancedMesh also sounds interesting, although the size of the sphere limits character space.

Wrt projection: I have an instancedMesh with sphereGeometries at random 3D coordinates. I believe that Vector3.project(camera) allows a 3D (x,y,z) coordinate to be projected to a 2D (sx, sy) screen coordinate. I would then use that 2D screen coordinate to draw the text on an html <canvas> which has the same viewport dimensions as the R3F Canvas but has a higher z-index, using context.fillText(text,sx,sy).
The hope is that the text is redrawn on the <canvas> after each time the user has interacted with the instancedMesh

7

I am able to get 60fps with my instancedMesh of 10,000 sphereGeometries.

The following Drei text implementation gives me 4fps (10,000 calls) for 10,000 labels.

function Word({ children, ...props }) {
    const fontProps = { color: 'white', font: '/Inter-Bold.woff', fontSize: 2.5, letterSpacing: -0.05, lineHeight: 1, 'material-toneMapped': false }

    return (
        <Billboard {...props}>
            <Text {...fontProps} children={children} />
        </Billboard>
    )
}

function Cloud({ count = 4, radius = 20 }) {
    // Create a count x count random words with spherical distribution
    const words = useMemo(() => {
        const temp = []
        const spherical = new THREE.Spherical()
        const phiSpan = Math.PI / (count + 1)
        const thetaSpan = (Math.PI * 2) / count
        for (let i = 1; i < count + 1; i++)
            for (let j = 0; j < count; j++) temp.push([new THREE.Vector3().setFromSpherical(spherical.set(radius, phiSpan * i, thetaSpan * j)), 'abcd'])
        return temp
    }, [count, radius])
    return words.map(([pos, word], index) => <Word key={index} position={pos} children={word} />)
}

                <Suspense fallback={null}>
                    <group rotation={[10, 10.5, 10]}>
                        <Cloud count={100} radius={20} />
                    </group>
                </Suspense>

10,000 labels would be an absolute maximum performance limit.
1,000 labels would be a more practical upper limit. With 1,000 labels I get 30fps and 971 calls.