Drag raycast jump with two cameras (different FOV)

Questions
1

Hi, I have a Three.js scene with two cameras:

  • camA renders a 3D mesh (target surface)

  • camB renders a plane overlay (UI/overlay object)

  • Cameras have different FOV, and both are correct visually for their own layer.

Drag logic

  • On pointerdown: raycast the plane with camB to start drag.

  • On pointermove: raycast the mesh with camA to get a world hit point and set plane position to that point.

Problem

  • On the first move, the plane jumps/teleports (wrong initial offset).

  • After that it “kind of” follows, but the grab point is inconsistent.

  • I think it’s because the same screen pointer corresponds to different rays / different NDC mapping between camA and camB.

Question

What is the correct approach to drag an overlay object when picking is in camB but placement is on mesh hit in camA, with different FOV?

Is there a standard way to map pointer between the cameras (screen-space offset, solving for equivalent NDC, etc.)?

2

I haven’t really done this with 2 cameras.. but with a single camera.. I usually do the initial raycast.. create a world space plane at the collision point.. orient it along the camera’s look axis.. with .lookAt … then raycast against that plane to track movement, and place indicators.
and instead of dragging by setting the position to the intersect.. I take the first hit point.. then on mousemove, raycast again, take the difference between start point, and current point, and move the object by that amount.