I have a server that can render a 3D scene into 2 flat images (each image from the view point of one of the eyes[^1])[^2]
These images are send to the javascript client via a WebSocket[^3].
What I am trying to achieve is the following:
Use three.js to simply display the images I receive in a way that a stereoscopic view is formed[^4]
Use three.js to obtain the camera view point for each eye so that I can send this information over the WebSocket for the server that is doing the rendering.
For (2), I think I have an idea on how to achieve that, however after going through the docs I am having problems to tackle (1). I have no idea from where to start to be honest.
Does anyone have any pointers to help with that? Any information on how to tackle pre-rendered scenes with three.js (or raw WebXR, WebGL APIs) would be extremely helpful.
[^1]: So that a 3D stereoscopic view can be created by using a VR headset
[^2]: Optionally, I can also change the server to concatenate the 2 images - corresponding to each eye’s view point - horizontally into a single image.
[^3]: In the future I plan to optimise this to use a WebRTC channel, but for now a WebSocket will do.
[^4]: I suppose that to achieve that I need copy the given images to the 2 halves of the rendering buffer.
you can query the XR stuff in threejs to find out which eye is currently being rendered, and switch your frame based on that.
that should be pretty straightforward… streaming camera.position and camera.quaternion…
I’m pretty skeptical about what kind of framerate/resolution/latency you’ll get rendering on the server and streaming to a client? WebRTC video might help… it’s also gonna tax the hell out of your server… but if you only need the server rendering a single user, i suppose it might work? Is there a reason you can’t render on the client?
yeah… video streaming has pretty high latency. It doesn’t matter for prerendered 3d video since the viewpoint change is all done on the client… but sending camera position to server, waiting for server to render, server compresses frame… streams to client… client renders… sounds like it might induce a lot of vr sickness?
Hi @manthrax, thank you very much for the pointers. Please find some comments below:
The main reason for the rendering not happening on the client is because the server was already written (by a different team of people). So we already have working software that does the image rendering using specialised hardware and that cannot be trivially ported to run over WebXR-own API + Javascript.
I do share the concerns about VR sickness and QoEs, that is something that we have to pay attention and be mindful of. For now the plan is to implement the system in a client/server manner and then run a full performance test to do a data-driven assessment on this topic.
Do you have any reference on how to copy 2 given images (one for each eye) directly into the rendering frame for the VR using three.js? That is my main pain point for the moment, I don’t know which APIs I should use to achieve that… The scene object does not seem to have APIs for pre-rendered stereoscopic frames…