Hi there,
I’m having trouble replicating the properties of a live camera on a PerspectiveCamera such that a ThreeJS 3D scene aligns as close to perfect as the scene from an undistorted camera feed from OpenCV.
I’ve gone through the traditional OpenCV calibration process with a checkerboard pattern, applied undistortion, and derived the camera intrinsic and extrinsic (mtx, dist, tvec, rvec). Just from the OpenCV projectPoint function, I’m able to overlay 3D points at all the expected positions just fine.
However, it’s not very clear to me on how to translate this to ThreeJS. I tried to derive the perspective projection matrix and sub that directly into the PerspectiveCamera projectionMatrix along with setting the camera position and quaternion.
The result weren’t amazing, comparing the 3D scene with the undistorted feed. There seemed to be some translation offset, and 3D elements positioned away from the centre of view are still not aligned. I guess one thing that wasn’t account for was the distortion coefficients as well - some posts I’ve looked at suggested using a custom renderer to apply dist from OpenCV?
If anyone has any advice to the proper method on how to match up ThreeJS perspective camera onto OpenCV undistorted camera feeds, that would be much appreciated!