Aligning DOM annotations with 3D objects in immersive-ar

Hello everybody,

I’m trying to place clickable annotations made with DOM elements on objects in the 3d scene.
It works for the normal “non-ar” view, but as soon as I start the AR mode (by clicking on one of the circles) they are placed slightly above the red/blue spheres in the corners of the cube and do not align perfectly like in the desktop view.

A demo can be found here: https://webxr-annotations.vercel.app/
The source code is here: https://github.com/LucaJunge/webxr-annotations

It seems to get worse if the cube is placed in the distance and the annotations are above the actual spheres.
This also happens with a raycaster (It is also visible in the demo, just click somewhere to update the ray)

Here are the things I tried so far:

  • Adjusting the renderer size based on the BoundingClientRect
  • Setting the fov to 50, so the perspective camera and the renderer.xr camera are the same
  • Adjusting the canvas size based on the window.innerWidth/Height

If someone has an idea, feel free to get in touch with me