Is there anything in JavaScript that can let me detect surfaces, so that I can render objects on them for AR experiences?
Since this question is not directly related to three.js, you might have more success when asking it at stackoverflow.
1 Like
I figured that of all groups of WebGL programmers in the web someone in the Three.js community would be most likely to know.
Plus I want to render stuff in Three, so there will be intricacies related to that, but maybe those would be separate questions. On the other hand, maybe some people would be interested to know about this for use in their Three projects?
3 Likes
Okay, sounds valid 
Itās the idea of an augmented reality API to provide the possibility to spatially register virtual objects in the real world. I donāt think you can emulate this with a simple video stream and some JS code. I mean if you want to work without markers but with arbitrary surfaces.
There is the WebXR spec that is available in Chrome Canary and should be coming soon to Chrome on mobile.
Docs:
Samples:
It doesnāt quite work the same as ARCore/Kit, instead of detecting planes you can use the Hit Test API to raycast against the world for your anchors.
As it is in Canary at the moment, it can potentially break at any update so I usually prevent auto-updates on it when I get a working version. Also its unlikely to be supported on iOS because they are pushing for AR QuickLook instead.
1 Like
Why not? Would it be too slow in JS?
Interesting! So if I wanted to detect a surface, Iād have to do a raycast sweep across the whole view and generate planes from those points? (it seems wasteful if the underlying engine already has that info)
I donāt know the exact motivations behind why its using hit tests rather than planes. It works more like it detects the mesh of the environment and casts against it. I believe they donāt give you direct access to the mesh for security reasons, which may limit the amount of hit tests you can perform as well.
The spec/system is still being worked on, so this has the potential to change over time.
What use case do you specifically want planes for? The hit test works well for placing items in a room but planes would be better if you were substituting carpets/tiles/paint etcā¦
It would defeat the purpose if we canāt reliably render things within boundaries of a surface (f.e. water falling off the edge of a table).
I am looking to use web tech (Three.js) to create a demo over the next couple weeks, and reliably detect surfaces and place objects. From the videos I see, looks like thatās possible, so Iāll tinker with it.
So far Iām having no luck getting any WebXR demos working in Chrome Android. Iāve enabled every flag that has āWebXRā in it. Maybe thereās API updates and the demos are outdated, but I havenāt checked the console for errors yet.
Any thoughts on WebXRForArCore? Looks like it isnāt updated much lately.
Its not ideal, but you can get the user to point to the boundaries. As far as Iām aware you canāt just detect a surface yet.
I use Chrome Canary with WebXR and WebXR HitTest flags turned on, and WebVR turned off: https://play.google.com/store/apps/details?id=com.chrome.canary&hl=en_AU
Chrome Android is still a little way off still.
WebXRForArCore was one of the first implementations and I donāt think its been maintained as actively.
1 Like
I think so. All this stuff mentioned in the ARCore documentation like motion tracking, environmental understanding and light estimation is not something you want to implement in JS for performance reasons.
1 Like
3D reconstruction from real-time stereoscopic images using GPU. Hmmmm, maybe it can be adapted for WebGL. 
@calrk Thanks, Chrome Canary worked!
mmm, hereās a much newer GPU-based 3D reconstruction paper. Looks like we can do it in a worker with OffscreenCanvas by transferring the webcam feed to a worker.
Hey @trusktr did you end up being able to detect surfaces with three.js?
In the meanwhile, there is an official example for the WebXR Hit Test API.
2 Likes
Didnāt get to try it yet.
Neat!
Somehow related: Iāve looked into PTAM to achieve this. More precisely into this GitHub - williammc/ptam_plus: PTAM (Parallel Tracking and Mapping) re-released under GPLv3.
And compiled it to JS using Emscripten. Now I would need to write JS bindings and see if it works.
There is also the start of a JS implementation here: GitHub - hitsthings/ptamjs: WIP!! Doesn't work yet (WIP and not functional)
Would be great to push forward on this 
2 Likes
Hey I found this, itās pretty interesting, but Iām not sure how they achieve it:
I tried to search around with the āSLAMā keyword, however what I found is beyond my knowledge 
just post it here to see if someone can give a clueā¦
yes! well, sort of .
(keep in mind im definitely not a pro in the field)
But yes there are tools that detect certain features of congruent pixels colors, and certain movements (opencv js , boofcv, are good places to start)
but theres no one overall solution, as far as i can tell from my long searching.
youāve got to bridge the components together to fit your specific needs
i think all thats really needed is
- detecting sobel or canny edges.
- Hough line detection
3 optic flow or motion detection capabilities,
so, find and reinforce lines in your field of view,
then group and track then. if they move consistently , reinforce that group and get a motion vector on it.
generally speaking, as you move in a direction, objects closer to you, will cross your field of vision quicker than objects far away from you.
Iām still thinking theres a method to resolve planes quicker than by tracking line group movements ā¦
im not sure on the method to derive distance from motion parallax.
and i believe these methods iāve outlined are called āstructure from motionā and ādense planar SLAMā
microsoft kinect fusion using the same methods,
i really think the depth mapping only helped refine the small details of structures, but the over all realization of the enviroment is created by joining the over mentioned methods.
Iām working on combining three scripts right now regarding this matter, and found this post researching some snags.
iād love to share what im working on and if you can give me maybe a day ish, Iāll post what i have for you.
I canāt believe there isnāt a js solution for all this.
yeah, it feels a bit heavy and slow being processed, but still.
2 Likes
Will be awesome to see! I think it would be interesting to port a solution to AssemblyScript (TypeScript that compiles to WebAssembly).
i am so sorry. ive been so busy with things and everytime i sit down to progress my coding i find myself studying more and more efficient options that i would rather build off of now instead refining later. but i do want to add more detail to my post.
I love three.js. but it hasnt offered my many out of the box (or close to it) solutions to meet arcore abilitiesā¦ but.
1 it has been very useful in building āholoportationā point clouds from colored video depth map streams i recieve of users. Add some octree optimization and im happy. (still need to pose estimate rigs, and fuse them, for each video/angle i receive, for a full 360 point cloud of the users.) but threejs handles that great!.
- i prefer three.js by itself for handling the stereo VR through the vrbutton, but AFrame gives me clear GPS data, ARtags, a little more structure, and using SuperHands along with it pretty much covers Physics, and manipulation of objects.
i find AFrame glitchy though, a little slow, and hinders my access to three.js at times. AND i had to create my own PowerOfTwo canvas video shader in three.js to achieve SeeThruAR for AFrame.
3.(oh boy. smh. gulp)
so, ARcore uses Motion/Point tracking (optic flow?)ā¦ and measures how fast it sees pixels move across the screen, in relation to each other. [if you move sideways, and look ahead, objects FAR way, appear to move SLOWER THAN ones that are CLOSER. This is called Motion Parallax.
i believe it finds and ,groups points together, that appear to be moving at similar rates. Then begins to build depth positions for each group according to the different rates.
then also gets an (1) overall average pixel speed on screen and (2) flow direction.
which it checks against (3) Accelerometer and (4) Gyro Data .
By themselves the 4 are not accurate enough, but when they are used together (aka Sensor Fusion) you can begin to plot your xyz location pretty accurately, and get which angle your camera is facing (pose estimation).
three/js offers some vector and matrice maths that can be useful in working all that out BUTā¦
OpenCV is pretty much the standard for point/motion tracking. focused on c++ they also have some java and js functions.
AND THEN,(new to me) thereās JSFEAT purely focused on JS Computer Vision.
Iāll just jump right in here and let you check out this example . be sure to his train_pattern and if im not mistakened youāre give a refrence frame(big green square) of your location when you trained. (moving the actual camera around should bring it into view)
I do think that can be considered a solid first step for JS based SelfLocalizationAndMapping all in one little script (Iāve looked forever for this. now i want to combine that with optimized canny edge detection, cause its appears a bit slow when working with the full video frame)
https://inspirit.github.io/jsfeat/sample_orb.html
and it has a RANSAC motion estimator function and all other kinda goodies.
really, if you have fairly stable zdepth readings, xyz and pose estimation with a dependable refrence frame ln spacetime like that ā¦ i do believe you could do most of what ARcore does.
-
Enviromental Understanding.
arcore does attempt to find thinks like planar surfaces and squares. like a wall, or table top, and i suspect object detection. then say, if it recognizes a chair, or a table, it will roughly attempt to put a corresponding , basic model in that place. that way, it can interact with it virtually. like it could place a VR pot of gold on your table, and say ālook!!! on the table. its gold!!!ā then POOF!! , the golds gone, and you could be amazed by ARcores meaness and question if it should remain on your phone or not after that.
ā¦ because instead you could just use this free web based JSFEAT equivalent without the hassle and the tricks. -
its aware of lighting, angles and shadows and attempts to rebuild an HDRI skymap light equivilant. how? not sure. probably like reversed ātexture bakingā?, maybe checks for brightest pixels in 360 within the scene then builds another pot of gold, and takes a guess at lighting it, then see if shadows and angles macth up as predicted? im not sure, but im guessing, weāre ok for now on that.
in conclusion::
point tracking
motion tracking
accel gyro data
ORB tracking,training
canny edge detection
optimize optimize optimize
JSFeat (openCV?)
AFrame
SuperHands
and of course.
THREE.js (its always been the glue that binds for me. <3)
(ā¦ oh, everything will disapear once its outta view, unless you stich scans together with āpoint registration or ICPā and store that point cloud data somewhere quickly and freely accessible)
but baby steps.
I donāt normally post in forums. like everā¦ ever.
i donāt know if im the bull in the china shop today, i donāt know if i hurt feelings or broke any belongings. im really just a noob, with a big, passion. i drempt of this stuff when i was 5 and 40 years later im realizing. its finally here, and i CAN actually play a part in it
and ! with JS
and THREE.js
thank you for tolerating me.
1 last questionā¦
Mugen87ā¦
i see youā¦ pretty much every day as i search the net for answers to three.js aframe p5.js, opencv , and ā¦ the weather.
and your always answering questions. like, ALL of them.
even mine,
because someone else asked it and you answered.
Soā¦ like, who are you exactly?
is this a casual, or professional passion?
do you own Mr,Doob?
is anybody FORCING you to answer these questions against your will?
(and some day, i will share my acheivments here, soon. im shy, and still workn)
<3
1 Like