Webxr Mirror Screen onto canvas

Questions
1

Hey, im building a Webxr app, i wonder how i could mirror what we see on VR into the computer screen,
i looked at old posts but the solution proposed does not seem to work anymore.

2

three.js does not support a spectator mode right now.

However, it should be supported by WebXR. There is a respective example however I have not tested it:

https://immersive-web.github.io/webxr-samples/spectator-mode.html

3

yes but isnt it a workaround? if not i d have to rewrite everything using raw webgl… its a pretty complex app that i build

4

Hi Lucy,

There is a work around, and it’s not very difficult.

You’ll want to create a separate camera for your spectator, so you don’t have the multiple views renderer. In your animation frame callback, after you render the scene normally, you’ll want to conditionally render again.

// render as normal
renderer.render(scene, camera);

// ... then conditionally render the spectator view
if (renderer.xr.isPresenting
    // we don't have a good check for whether or not mirroring is possible
    && isDesktop()
    // Firefox is still on the old, prestandardized WebVR API, so it can't do this
    && !isFirefox()) {

    // Copy the XR Camera's position and rotation, but use your
    // main camera's projection matrix
    const xrCam = renderer.xr.getCamera(camera);
    spectator.projectionMatrix.copy(camera);
    spectator.position.copy(xrCam.position);
    spectator.quaternion.copy(xrCam.quaternion);

    // we'll restore this later
    const currentRenderTarget = renderer.getRenderTarget();

    // turn off the WebXR rendering
    renderer.xr.isPresenting = false;

    // render to the canvas on our main display
    renderer.setRenderTarget(null);
    renderer.render(scene, spectator);

    // reset back to enable WebXR
    renderer.setRenderTarget(currentRenderTarget);
    renderer.xr.isPresenting = true;
}
2 Likes
5

well actually i did just code my own graphic library to fix it lol so its allright, just i need a good gltf importer and its a bit complicated but its on its way.

Already have,
Webxr, with controllers that shows up and all with buttons that move when pressing them,
Text with some sdf shader that display glyphs,
it runs on electron, cause i needed it to be on desktop,
A basic gltf importer,

and what next to come:
a full gltf importer, i got this to work in my program:
KhronosGroup/glTF-Sample-Viewer: Physically-Based Rendering in glTF 2.0 using WebGL (github.com)

but its not really my code i just hack around to call its render during my renders, so i need to refactor and stuff.

6

Hi capnmidnight, your piece of code has almost worked for me on three 148!

The only change I’ve made was:
spectator.projectionMatrix.copy(camera);
to
spectator.projectionMatrix.copy(camera.projectionMatrix);

4 Likes
7 
import React, { useRef, useEffect } from 'react';
import { Canvas, useFrame, useThree } from '@react-three/fiber';
import { isDesktop, isFirefox } from './yourUtilityFunctions'; // You need to implement these

export const SpectatorView = function({ children }) {
  const { gl, camera, scene } = useThree();
  const spectatorCameraRef = useRef();
  const originalRenderTarget = gl.getRenderTarget();

  useFrame(() => {
    if (gl.xr.isPresenting && isDesktop() && !isFirefox()) {
      const xrCam = gl.xr.getCamera(camera);
      const spectatorCam = spectatorCameraRef.current;
      spectatorCam.projectionMatrix.copy(camera.projectionMatrix);
      spectatorCam.position.copy(xrCam.position);
      spectatorCam.quaternion.copy(xrCam.quaternion);

      // Temporarily disable WebXR rendering.
      gl.xr.enabled = false;
      gl.setRenderTarget(null); // Render to the canvas
      gl.render(scene, spectatorCam);

      // Restore WebXR.
      gl.setRenderTarget(originalRenderTarget);
      gl.xr.enabled = true;
    }
  });

  return <perspectiveCamera ref={spectatorCameraRef} />;
}