I am working on an AR browser based project. I went through threejs sample examples and I rendered a cube on top of a html video. Now I want to move the cube according to the projection matrix I receive from the opencv functions. How can I set three js camera and 3d cube to work accordingly ?
Please provide any direction, any sample or something would be very helpful.
I guess you would start by decomposing the OpenCV projection matrix using the decomposeProjectionMatrix function.
https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#gaaae5a7899faa1ffdf268cd9088940248
This will give you the camera’s rotation, translation, and something called a 3x3 camera intrinsic matrix.
Next step - convert the rotation and translation to the three.js coordinate system (I’m not familiar with OpenCV, this may be easy or hard) and then apply them to the camera. Then investigate whether the 3x3 matrix contains data you can use to set the three.js perspective or orthographic camera (like near, far for all cameras and then aspect, fov for the perpective, or left, right, top, bottom for the orthographic).
I did find some info on the 3x3 matrix elsewhere on the docs page I linked. You’ll need to investigate further to see what fx, fy, cx, cy are and whether it’s possible to map them to three.js cameras.
Hey, did you find a solution?
Yes I found the solution by trial and error but I moved to WebXR since it suffice my use case. So this issue is now outdated.
I’ve written a tutorial on this topic (with code examples). Check it out! https://segments.ai/blog/simulating-cameras-three-js
