How to accurately align panoramic images with a 3D model at hotspot locations in Three.js?

I am working on creating a 360° virtual tour where panoramic images should seamlessly integrated with a 3D model.

Here’s the workflow: The user clicks a hotspot.(PRESS m key) The camera moves to the hotspot’s position in the 3D model. The view switches to a spherical panorama view where the user can freely rotate the camera in 360 degrees.

My requirement I want panorama image overlay on 3d model correctly.

What I have done

Created a sphere and applied texture(panorama image on that sphere) Sphere position set at hotspot position as this the place where panorama was capture.

function createSphereAndLoadPano() {
  let texturePath = 'https://i.imgur.com/wJeAbfa.jpeg';

  const textureLoader = new THREE.TextureLoader();
  const panoramaTexture = textureLoader.load(texturePath);

  const panoMaterial = new THREE.MeshBasicMaterial({ map: panoramaTexture });

  const sphereGeometry = new THREE.SphereGeometry(500, 60, 40);
  sphereGeometry.scale(-1, 1, 1);
  const panoSphere = new THREE.Mesh(sphereGeometry, panoMaterial);

  panoSphere.up.set(0, 0, 1);
  panoSphere.rotateX(Math.PI / 2);
  let rad = THREE.MathUtils.degToRad(90);
  panoSphere.rotateY(rad);
  let vecPos = new THREE.Vector3(101.02, 97.17, 4.35);

  panoSphere.position.copy(vecPos);

  scene.add(panoSphere);
}

Here’s the live demo :Demo Press m key to move and activate the pano mode.

I have done hard coded manipulation to rotate the sphere to set alignment but this does not work well

I want to know.

What are the standard techniques and considerations for accurately aligning 360° panoramic images with a 3D model at specific hotspot locations in a virtual tour?

Like these demos?

1 Like

What are the standard techniques and considerations for accurately aligning 360° panoramic images with a 3D model at specific hotspot locations in a virtual tour?

In order to perfectly align a 3d model with a panoramic image you have to know both the position and the orientation of the image in a common 3d coordinate space with the 3d model in the first place. If you don’t know them in advance you will have to infer this piece of data, and keep in mind that this should be consistent to the 3d model origin (again, you have to know all of this to be able to fit them).

Some pictures:

  1. I tried eyeballing the position of the camera just by changing your values, the preview shows that the 3d model is overlaying the structure just fine (you don’t see the structure behind):

  1. Regarding camera orientation, you can see that rotating -90° is not enough, since the original photograph is not perpendicular with the structure in the first place. I just added 0.3, but you should find this value properly in order to better align the image (you can see the vertical yellow element to the right is not perfect)

As a suggestion, you can better inspect the model and find those values directly in i.e. blender

I Hope this shed some light on the topic

1 Like

I did also look into this, but didn’t come to a conclusion and hence didn’t post an answer. As an extension to the very good answer from @Antonio, I’d like to add the following:

The utmost importance of replicating the exact same viewing position which the camera had when the spherical panorama was taken cannot be overstated. I saw the shadow of the tripod in a certain direction. Maybe you can “carry over” the intersection of two reference lines from reality into your model.

Replication of camera position includes its height also, so account for the tripod height, too. It’s also advisable, to use the same focal lens in your virtual Three.s camera as the physical lens, which was used when taking the shots for the spherical panorama, to avoid parallax aberrations as much as possible. These would be most pronounced for objects in the foregrund - the closer those are, the worse it would be.

Last, I had good results putting some vertical straight lines into a copy of the equirectangular bitmap at the 0°, 90°, 180°, 270° longitude as a visual debugging aid (at leftmost edge, 1/4 , 2/4, 3/4 image width. These give you a nice crosshairs at the nadir, which will help with horizontal positioning. Example:

Spherical Panorama using the above bitmap:

3 Likes

Exactly, very good observation!
I forgot to mention, I started setting the camera fov as 60°, but then ended up further adjusting it with the mouse scroll and a custom function until I found a close-enough value.

If the scene has known/regular geometry (like indoor and/or regular urban configurations), there are workarounds to find this values after the image is captured. This is not the case with nature or irregular shaped scenes, @vielzutun.ch suggestion of adding straight lines is an interesting method to try :+1:

From time to time, when I have the chance to, I usually capture around 5 images more, at different positions, to process them with photogrammetry. This would give you the relative position and orientation of the camera of interest very precisely; later, you just need to find a set of points in space that matches the locations in the 3d model, which can be solved by using an algorithm for minimizing the difference between the two sets of points, like ICP.

1 Like

Like these demos?

If that’s what you’re after, you’re gonna need a depth map to go along with your images (or if you know exaxctly the location where a photo was taken relative to the 3D models, then you can use that to make a depth map).

Once you have a depth map, you can morph the image in a shader, or as a geometry.

Here’s an example pen using Lume HTML (HTML elements powered by Three.js) to apply a color map and depth map to a plane, making it 3D:

Here’s another project that does the same thing in plain Three.js:

Demo:

https://thygate.github.io/depthmap-viewer-three/

Here’s another project that does the depth effect all in the fragment shader (faster because it avoids large polygon count, but also potentially less accurate):

Demo:

The only difference between those is you will be doing it on a sphere geometry (or spherical in a fragment shader) for a 360 image.

Here’s an official Threejs 360 example!

https://threejs.org/examples/webxr_vr_panorama_depth.html

While we’re having some fun, how about some more examples (not necessarily Threejs, but same idea):

Once you get that far (images + depth map) then you will need to write code that will transition between two of those, which is probably the easy part, icing on the cake (the piece everyone is currently charging money for):

  • begin fading out the current 360 depth image out
  • begin animating towards the new position of the new sphere geometry (or shift the perspective of a fragment shader to achieve similar effect)
  • begin fading in the new 360 depth image

I’m interested in this myself so I figured I may as well paste of a bunch of examples for ideation.

Hope that paints the picture!

1 Like

Blockquote in order to perfectly align a 3d model with a panoramic image you have to know both the position and the orientation of the image in a common 3d coordinate space

First of all thanks for going through the question and code and giving your input much appreciated.

Just to confirm, when you mention the position and orientation of the image, you’re referring to the camera’s position and orientation at the time the panorama was captured, right? So, the image itself doesn’t have these properties, but to align it correctly with the 3D model, I need to position the panorama sphere at the original camera’s location (position) and rotate it to match the camera’s facing direction (orientation).

also when you mention a common 3D coordinate space, do you mean that both the 3D model and the panoramic images (or rather, the camera position and orientation when the panorama was captured) should reference the same origin and axis directions?

Blockquote Regarding camera orientation, you can see that rotating -90° is not enough, since the original photograph is not perpendicular with the structure in the first place. I just added 0.3, but you should find this value properly in order to better align the image (you can see the vertical yellow element to the right is not perfect)

The -90° rotation is a hardcoded value that I arrived at after trial and error to reduce the visible offset between the 3D model and the panorama. It’s not a generic solution—it was manually adjusted based on picking a reference point in the 3D model and rotating the sphere until the alignment looked close enough.

I understand this isn’t scalable for multiple hotspots, which is why I’m exploring a more automated approach. Here’s what I’ve tried so far:

  • I have point cloud data (in E57 format) of the 3D model, and the same scanner camera was used to capture the panoramic images.
  • When I opened the E57 file in CloudCompare, I found the camera rotation metadata. I applied this rotation directly to the sphere, assuming it would eliminate the need for manual adjustments. However, even with that, some manual rotation is still required, which again isn’t a generic solution.

I’m now wondering if the issue isn’t just the sphere’s orientation but also related to the texture mapping itself. Specifically:

  • Could the misalignment be because the texture’s wrapping start point doesn’t match the camera’s original capture starting point?
  • Even if the sphere is oriented correctly, if the panoramic image is wrapped starting from the wrong reference point, wouldn’t that cause the same offset issues?

Blockquote As a suggestion, you can better inspect the model and find those values directly in i.e. blender

Sorry I did not get you here. can you please elaborate.

Apologies for the long message, and also for any language mistakes or if I didn’t explain things clearly. I appreciate your patience and insights!

Blockquote Like these demos?

@trusktr

Yes Exactly I want to achieve similar to this !!!

I’ve ensured that the camera position is exactly the same as when the spherical panorama was taken. Could you please elaborate more on the part about “carrying over” the intersection of two reference lines from reality into the model?

I did try setting the same FOV for the virtual camera, but unfortunately, I didn’t achieve the desired results.

That’s correct

That’s correct

Yes, it is expected to be like this. Remember that this transforms correspond to local space of the scanner -not global- so even by activating gps sensor will give you “reference values” of position and orientation.

Most definitely. You can think of this as in top view (regarding your -Z axis, as you are working Z up): to be able to work consistently across your data, all the captured images should aim at the same azimuthal direction. If not, it is a good idea to cancel this local rotations in advance. Google Street View does store and load local rotation of each pano, but this requires a dedicated DB, so it’s not impossible tho.

I was simple pointing out that by using a DCC software that allow you to overlay the pano to the 3d model, like Blender, you could find those values with certain confidence, and that applies to all of them: position, orientation and FOV of the camera, and position of the 3d model’s origin point, all in a shared 3d coordinates space.

This gives a more clear objective. Keep in mind that still need the mentioned values. If you were rendering the panorama images from a 3d model, this could be really easy. As the images are captured from the real world, you need to consider a solid strategy to acquire or infer them.

2 Likes

I’ve ensured that the camera position is exactly the same as when the spherical panorama was taken.

You didn’t succeed in that. I uploaded a modified image with vertical lines in it which clearly shows where the nadir of the panoramic image is: at the center of the big black circle

Please change in your Demo (1st post) the following line to try out the modified image:

function createSphereAndLoadPano() {
  let texturePath = 'https://i.ibb.co/v6PBTJXm/Bohrinsel-Lines.jpg';

Looking straight down (OrbitControls @.minPolarAngle), you see the following scene:

I conclude, that the virtual camera is at least 1 meter off laterally, probably more. Those auxiliary red lines (starting at the leftmost pixel column!) also give you a hint as to how much the orientation of the equirectangular bitmap is different from the coordinate system used in your 3D model, assuming the latter is axis-aligned.

1 Like

In the physical world, aim for over two points, preferrably having a large distance between them, which you can easily identify /replicate in your 3D model. You can use a standard (cheap) laserpointer for that.
Using a second laserpointer, repeat the first step using a direction perpendicular to the first. Doesn’t have to be overly exact. Important is to have a clear point of intersection.

Attach a plumb line to the central axis of your tripod, which is going to take the panoramic image. Move the tripod around until both laser beams hit the plumb line at the same time.

In the virtual world, place your virtual camera (in the horizontal plane) at the intersection of those two “aim for over two points” lines.

2 Likes

Ohh, I got your point now about drawing vertical lines for visual cues. Actually, I got this position from Leica Cyclone software and didn’t cross-check it, as Leica cameras/scanners were used to capture the data.

When you say “the orientation of the equirectangular bitmap is different from the coordinate system,” do you mean the orientation of the sphere on which the panorama is applied is different?

If the position were correct, I should see the alignment like in your demo when looking straight down, correct? like this.

yes - it’s about the rotation about the sphere’s vertical axis.

yes.

I propose the following sequence of adjustments:

  1. Place your virtual camera at horizontal origin and proper height.EDIT: place your virtual camera at the horizontal point corresponding to the intersection of the two aim for over two points lines from the physical reality.

  2. In your combined view (background + virtual camera + model), looking straight down, shift your model including camera(!!) laterally, until coordinate origins of reality and virtual reality match. Maybe temporarily make the floor element of your model invisible or semi-transparent for this step, to maintain a view of the background crosshairs.

  3. Re-orientate background sphere so crosshairs become horizontal/vertical (again, assuming your model is axis-aligned. EDIT: AND your panoramic image starts with an axis alignment, too!).

  4. Scale as needed.

1 Like

@Antonio @trusktr @Antonio First of all thanks for the inputs you guys have provided I am working on pointers/leads that you guys have provided. I achieved something close to what I want to achieve will update demo. and keep you in loop

3 Likes