Failed attempt to replicate Threebox CameraSync

Hello Three.js world,

Maybe you’ve heard of Threebox (also this fork).

We have a need to basically do this on our project, however our application is so far built on @react-three/fiber and other libraries e.g. @use-gesture/react for gesture handling.

I’ve been able to (with a bit of help from the R3F Discord) render a basic R3F box mesh onto the same canvas that Mapbox renders to with the code here: r3f-mapbox-integration-xp/src at 38307d7c8ec69287ec6058c2a1a3d402c8368900 · mysterybear/r3f-mapbox-integration-xp · GitHub

In the store.ts file I wrote a hook to return functions to either enable or disable all of Mapbox’s interactions, so that any interaction code on our Three side can disable-before and enable-after any interaction (similar to how camera/orbit controls are sometimes managed).

So now I think the final step needed is to sync the cameras! There’s a separate branch here where I’ve tried to copy the Threebox camera syncing code into a hook: r3f-mapbox-integration-xp/cameraSync.ts at bb988fe73b4a7d99ecce7e57c1b546c6776b8f43 · mysterybear/r3f-mapbox-integration-xp · GitHub

I had to make some changes because the old code is class-based javascript and also what appeared to be redundant (some unused vars and the setup function seemed redundant because it just calls update anyway). I’m pretty sure it should work the same as threebox/CameraSync.js at d2a5742a454d30dd2295a892e2e65cc1a41935b8 · jscastro76/threebox · GitHub

Sorry, I don’t actually have a specific question, it’s more of a “well, it doesn’t work… What should I do?” I’ve not worked with projection and transformation matrices at this level before I’m afraid!

Any help or direction appreciated!

I think I’ve thought of a way to debug this, for starters I can disable rendering the Mapbox frames and I can check that the box is rendering at the coordinates we’re expecting (I think we’ve turned our Three.js units into either Mercator or long/lat, will have to check). Then I should also be able to use camera helpers to debug the camera I guess.

Sorry for the noise, I will try this and then report back if still having issues.