I am making a electron app in which i wanted to showcase 360 degree image which are taken by insta360 camera and the extension of the picture are .insp
so when use the below code there is always a stitching issue in the 360 degree image.
import React, { useEffect, useRef, useState } from ‘react’;
import * as THREE from ‘three’;
import { OrbitControls } from ‘three/examples/jsm/controls/OrbitControls’;
const Image360Viewer = () => {
const canvasRef = useRef(null);
const [image, setImage] = useState<string | null>(null);
const handleImageUpload = (event: React.ChangeEvent<HTMLInputElement>) => {
const file = event.target.files?.[0];
if (file) {
const imageUrl = URL.createObjectURL(file);
setImage(imageUrl);
}
};
useEffect(() => {
if (image && canvasRef.current) {
const canvas = canvasRef.current;
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer({ canvas });
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);
const textureLoader = new THREE.TextureLoader();
textureLoader.load(image, (texture) => {
texture.minFilter = THREE.LinearFilter;
texture.wrapS = THREE.ClampToEdgeWrapping;
texture.wrapT = THREE.ClampToEdgeWrapping;
const geometry = new THREE.SphereGeometry(500, 60, 40);
geometry.scale(-1, 1, 1);
const material = new THREE.MeshBasicMaterial({ map: texture });
const mesh = new THREE.Mesh(geometry, material);
scene.add(mesh);
camera.position.set(0, 0, 0.1);
const controls = new OrbitControls(camera, renderer.domElement);
controls.enableZoom = true;
controls.enablePan = false;
const animate = () => {
requestAnimationFrame(animate);
controls.update();
renderer.render(scene, camera);
};
animate();
});
const onWindowResize = () => {
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize(window.innerWidth, window.innerHeight);
};
window.addEventListener('resize', onWindowResize);
return () => {
window.removeEventListener('resize', onWindowResize);
};
}
}, [image]);
return (
<div>
<input type="file" onChange={handleImageUpload} />
<canvas ref={canvasRef} style={{ width: '100%', height: '100vh' }}></canvas>
</div>
);
};
export default Image360Viewer;
i have attached the screenshot of what is happening can someone please help me to fix this bug and also if you thing that the image maybe corrupted then i tryied to open the image in the official insta360studio and it worked perfectly without any stitching issue.
if you want some more information i can provide you more information.
Code snippet looks allright-ish
Please share the actual image you are using, it looks like it could generated with a wrong projection
the image i am using is looking fine the the insta360 studio. the file that i am using is bigger than 8 MB it size is 25MB so can’t upload it here.
I’ve looked up the insta360 line of panoramic cameras. The cameras seem to have a distinct distance between the two lenses, back-to-back. So they need to cover a little more than a 180° hemisphere each, to arrive at seamless 360by180 panorama, even without allowing for an (IMHO: inevitable!) overlap between the two half-images.
I’ve done some extensive work on such spherical panoramas myself, using stock lenses for my Sony A9 full format camera.
See this and this, for example.
You need to provide sufficient overlap between any two adjacent shots to allow for a seamless tiling. The stitching is done in the dedicated commercial software PTGUI Pro, which comes with a library of settings for the most commonly used lenses that are used for spherical panoramas. As far as I remember, you can download a no-strings-attached version of this software for free. The only downside of this evaluation copy is, that each output comes with a visible watermark, which I think is fair enough.
That said:
The supported lenses distinguish each from one another by a set of specific parameters which are necessary for “reversing” the lense-specific distortion which is inevitable when capturing a hemisphere onto a flat 2D Bitmap.
Back to your insta360:
To me its obvious, that insta360’s lenses too need such a distortion-reversing algorithm to make it work seamlessly. It’s also just as obvious, that the software developers at insta360 get acces to this propriatery information, while you as an “external” developer don’t have this information.
This is not a Three.js problem. Rather a mathematical problem for which the necessary information is - alas - undisclosed.
The insta360 series of panoramic cameras is nice while it works. But shields end users from the necessary information and understanding what’s required behind all this, to make it work.
Exactly what I was suggesting, thanks @vielzutun.ch for such a detailed explanation
@Mohamad_Anas for us its not obvious the image you are using yo test your application. You need to make sure it is coming in equirectangular projection or other supported by threejs
A simple screenshot would be fine to share it here, as a workaround you can test it for yourself with an online 360 image viewer like this
@vielzutun.ch @Antonio Thanks for having a look into the problem but when i use the the image in an online 360 viewer the issue is same they also have these type of stitching issue in it.
when i open this file in insta 360 studio it works perfectly. so what i did is that i made a script in my app in which it opens the insta 360 studio and then imports the image and then export it as 360 photo in jpg format. and when i use that picture in the viewer then it works perfectly.
Is there some other solution you have that could work.
what i think could be the issue is the .insp file rendering because .insp image are not really supported in these library so maybe the image is the issue so i used the offcial software to convert the image into jpg
this is how the image looks like when exported from the official app. the size of this image is really big (35MB).
1 Like
Thanks for reporting back on this.
I think, you are using a really clever way of getting the benefit of undisclosed information, without actually having that information. Sort of a graphical pre-processing. I can’t think of a better way to do it.
I also checked out your attached image, using the free FSPViewer software. Although small and low-res, I couldn’t find a seam in your image except for a tiny glitch around the nadir, where the tripod is not perfectly blended
One additional piece of advice: you can/should place the line, where a seam might appear, strategically. Meaning, you should rotate the camera in such a way as to avoid the seamline to fall within a continuous shape, but rather between continuous shapes where they would be harder to discern.
Tested the image above with a small program with vanilla JS (sorry, I’m not reactive). The image is OK. I see no seams. The distortion at zenith and nadir are expected, they are caused by some conceptual issues of UV mapping on a sphere.
Here is what I see:
1 Like
Here is a fork of one of my R3F examples that uses the image you posted above, and with the renderers max anistropy.
2 Likes
@PavelBoytchev as seen in the video you have attached the image look fine because it is converted to .jpg form i have found a solution as said above. but in a normal 360 view my .inps image always creates a problem.
As said before, this is not a threejs problem.
Inps is a propietary format for storing insta360 sensor data, which apparently comes in dual fish projection (plus unknown overlapping and image space margin data among other unknown optical / lens / camera body info). This is not going to work ‘as-is’ in threejs unless you develop a bespoke piece of code to stitch and re-project from (unknow) dualfish to equirectangular.
So to recap, using instaStudio for converting to jpg is your best bet.
2 Likes
