Hi, new here but have been referred to this conversation by @ThorstenBux.
While by no means a computer vision expert, I am interested in this, primarily to enhanced location-based (i.e. using GPS and sensors) AR by detecting surfaces and thus allowing more realistic placement of objects so that they actually appear on the ground. My specific use-case is outdoor AR for navigation for walkers and hikers.
I have been working with Thorsten a little on this, and have investigated the PTAM library which he has suggested, but so far, I am encountering problems with using it from emscripten, which are not yet solved.
Thanks to this thread, Thorsten’s advice and a bit of research, I have some idea of the general procedure needed. We need point or edge detection, which seem to be fairly well established algorithms and can be done from JS libraries such as tracking.js or JSFeat. Once we have collections of points, we can then detect edges or planes (again, there seem to be well-defined algorithms to do this)
The difficulty is converting the 2D points to 3D coordinates, ready for rendering (for example with three.js). For this a camera pose estimation is needed, e.g. using solvePnP. However, to obtain this (from my rudimentary knowledge of this area) a calibration step is needed using known correspondences of 2D and 3D points (e.g. a paper marker of specific dimensions at a given distance from the camera).
This (i.e 2D to 3D point conversion) is the main stumbling block, I think. If the 2D-to-3D conversion is solved the rest of it, while not easy, at least looks well-defined and there are plenty of algorithms available to help us. PTAM looks like one possible approach but does anyone have any other suggestions?