3d geometry with 360 image!

Questions
,
1

Hello guys,
I am trying to do a project like this: https://rendertovr.com/player/?maindemo.
I have some questions:

  • Should I use cubemap or use textureloader as a sphere geometry?
  • In the example, the pointer changes according to the intersections and surfaces very precisely. So is there a way to get the surfaces and 3d geometry in there accurate with a 360-degree image?
    Many thanks
1 Like
2

  • In either case you want to wrap a surface with a texture, so for the cubeMap you will need a box geometry too. While both approaches are valid, in this scenario using an equirectangular projected image does offer a couple of benefits for computing this kind of smooth transition.

  • Regarding intersections, one way is to retrieve surface normal from a non visible proxy representation of your spatial world. This could be in world space (i.e. low-poly geo with a raycaster) or even in image space (surface normal encoded image from original hi-poly). Google street view, which partially inspired this kind of matterport transitions in the first place, managed to store depth data in a very compact and efficient way, all in image space.

This topic is still hot and further developments are being studied atm. Rendertovr developer commented advances in a previous post here, if you search it maybe you can find aditional implementations details.

3

Actually this looks very similar to Matterport, they use a depth camera they developed for this specifically to capture the full scene in 3D and use the 3D model for the transitions as well as that kind of cursor projection. Without you can’t archive this without some advanced technique reconstructing the depth as even advanced ones of today with AI relay on multiple captures from displaced positions.

In this case this also would be different as you got a cubemap, for the 3D transitions anyway you will need a full 3D scene composed by the reconstructed depth from all spots. Even though it is possible now to reconstruct depth from a single image, it might be not reliable enough for this or (if you find papers for it) might not be made for cubemaps.