[Solved] WebXR + Reflection => strange camera matrix

smarthug · October 7, 2021, 11:02am

This PR fixes this problem

github.com/mrdoob/three.js

WebXRManager: Fix XRCamera Local Space Behavior by Reverting #21964

mrdoob:dev ← zalo:revert-21964

opened 09:56PM - 18 Aug 21 UTC

zalo

+1 -4

This PR fixes a bug introduced in r130 by reverting how XR Camera calculations h…andle local space (by partially undoing #21964 (which appears to conflate `matrixWorld` and `matrix`)). This enables the XR camera to be parented to another moving object without odd behavior once again. If this isn't the correct fix; I'd be happy to test alternatives in the context of my application. Related issues: #21964 , https://github.com/google/model-viewer/pull/2464/files#r647866462

Hi all
I am working on WebXR teleportation.

Everything worked fine until I add water to the scene. (“three/examples/jsm/objects/Water.js”)

This is how teleportation works

let cameraRig = new THREE.Group();
cameraRig.add(camera);
scene.add(cameraRig);


function teleport() {
    cameraRig.position.add(new THREE.Vector3(10, 10, 10));

    console.log(`camera.matrix : ${cameraRig.children[0].matrix.elements}`);
    console.log(`camera.matrixWorld : ${cameraRig.children[0].matrixWorld.elements}`);
}

And this is what I get in console

camera.matrix : 1,0,0,0,0,1,0,0,0,0,1,0,10,11,10,1 
camera.matrixWorld : 1,0,0,0,0,1,0,0,0,0,1,0,20,21,20,1

What I should get is

camera.matrix : 1,0,0,0,0,1,0,0,0,0,1,0,0,1,0,1 
camera.matrixWorld : 1,0,0,0,0,1,0,0,0,0,1,0,10,11,10,1

Order to reproduce error

Enter XR
Press t to teleport and check console
Press w to toggle water (remove/add)
Press t to teleport and check console

I have tested it with
Win10 + Chrome + Webxr simulator plugin WebXR API Emulator - Chrome Web Store
Oculust Quest 2 + Oculus Browser

I have found a walk around but it feels unsafe. ( removing scene.add(cameraRig) )
I suspect Water’s onBeforeRender method.

Why does this thing happen? Will it be fixed?

smarthug · October 12, 2021, 5:48am

I assume some people might not comfortable with react, so I updated Codesandbox demo to plain javascript. And removed all dependencies other than three.js.

I guess the bug is caused because some of w3c webxr standards are not compatible with three.js ??

Will this problem vanish when webGPU and WGSL replace old regacies??

Any thoughts would be thankful!

trusktr · October 13, 2021, 5:09am

Did you test with native WebXR to make sure it is not a problem with the emulator extension? (I haven’t verified your bug yet)

smarthug · October 13, 2021, 7:55am

@trusktr
I have tested it with Oculus Quest2 + Oculus Browser

chrome://inspect/#devices

codesandbox example running and its console

I am quite sure this is not a bug from WebXR Emulator since I found this bug using Oculus Quest2 with Oculus Link

smarthug · October 13, 2021, 7:59am

I rounded the camera.matrix and the camera.matrixWorld in console for better readability.

A youtube video that shows how to produce the same error.

trusktr · October 13, 2021, 2:14pm

Seems like a “bug” in the design of Water.js. It is not accounting for how an end user may use a camera.

Here you can see it does a bunch of stuff with the camera that you pass into WebGLRenderer:

github.com

mrdoob/three.js/blob/d9af9952fc32c2a1281707946177135b90297aab/examples/js/objects/Water.js#L213

    
      
          			material.uniforms[ 'alpha' ].value = alpha;
          			material.uniforms[ 'time' ].value = time;
          			material.uniforms[ 'normalSampler' ].value = normalSampler;
          			material.uniforms[ 'sunColor' ].value = sunColor;
          			material.uniforms[ 'waterColor' ].value = waterColor;
          			material.uniforms[ 'sunDirection' ].value = sunDirection;
          			material.uniforms[ 'distortionScale' ].value = distortionScale;
          			material.uniforms[ 'eye' ].value = eye;
          			scope.material = material;
          
          
			scope.onBeforeRender = function ( renderer, scene, camera ) {
          
          
				mirrorWorldPosition.setFromMatrixPosition( scope.matrixWorld );
          				cameraWorldPosition.setFromMatrixPosition( camera.matrixWorld );
          				rotationMatrix.extractRotation( scope.matrixWorld );
          				normal.set( 0, 0, 1 );
          				normal.applyMatrix4( rotationMatrix );
          				view.subVectors( mirrorWorldPosition, cameraWorldPosition ); // Avoid rendering when mirror is facing away
          
          
				if ( view.dot( normal ) > 0 ) return;
          				view.reflect( normal ).negate();

Something is going on there.

smarthug · October 14, 2021, 1:58am

@trusktr
Thank you for the feedback.

I have played around with Water.js and I have found something.

If I comment out this line everything works fine

github.com

mrdoob/three.js/blob/d9af9952fc32c2a1281707946177135b90297aab/examples/js/objects/Water.js#L276

    
      
          				const currentShadowAutoUpdate = renderer.shadowMap.autoUpdate;
          				scope.visible = false;
          				renderer.xr.enabled = false; // Avoid camera modification and recursion
          
          
				renderer.shadowMap.autoUpdate = false; // Avoid re-computing shadows
          
          
				renderer.setRenderTarget( renderTarget );
          				renderer.state.buffers.depth.setMask( true ); // make sure the depth buffer is writable so it can be properly cleared, see #18897
          
          
				if ( renderer.autoClear === false ) renderer.clear();
          				renderer.render( scene, mirrorCamera );
          				scope.visible = true;
          				renderer.xr.enabled = currentXrEnabled;
          				renderer.shadowMap.autoUpdate = currentShadowAutoUpdate;
          				renderer.setRenderTarget( currentRenderTarget ); // Restore viewport
          
          
				const viewport = camera.viewport;
          
          
				if ( viewport !== undefined ) {
          
          
					renderer.state.viewport( viewport );

Water’s onBeforeRender method uses the camera to make water reflection. It seems it does not mutate the camera
This bug only occurs in XR, I suspect WebXRManager’s updateCamera function.

I will let you know If I find more.

smarthug · October 14, 2021, 5:02am

I have found that it is not only happening with Water.js, it also happens with Reflector.js (three/examples/jsm/objects/Reflector.js)

I changed codesandbox demo’s water to reflector

smarthug · October 14, 2021, 5:43am

I think I found the source of the bug.

In WebXRManger, updateCamera function gets called every render and it checks whether the camera has a parent.

Reflector’s onBeforeRender method renders reflection with a virtual camera. Maybe that virtual camera has something to do with updateCamera function.

github.com

mrdoob/three.js/blob/4afef617be65dcd67a67cfbf1bd5cb1864cef50b/src/renderers/webxr/WebXRManager.js#L454

    
      
          			const far2 = far + zOffset;
          			const left2 = left - xOffset;
          			const right2 = right + ( ipd - xOffset );
          			const top2 = topFov * far / far2 * near2;
          			const bottom2 = bottomFov * far / far2 * near2;
          
          
			camera.projectionMatrix.makePerspective( left2, right2, top2, bottom2, near2, far2 );
          
          
		}
          
          
		function updateCamera( camera, parent ) {
          
          
			if ( parent === null ) {
          
          
				camera.matrixWorld.copy( camera.matrix );
          
          
			} else {
          
          
				camera.matrixWorld.multiplyMatrices( parent.matrixWorld, camera.matrix );
          
          
			}

It is not clear to me why this behavior is needed and how to solve this problem.

trusktr · October 16, 2021, 11:06am

I think you’re into something. It seems like an oversight in how those example classes were designed and/or how WebXRManager is designed. I think this is worthy of a bug report on the GitHub repo.

smarthug · October 17, 2021, 12:54am

@trusktr
Thank you for the feedback.

After few more research I will create an issue on the Github repo.

I used to be Unreal Engine 4 Developer.
I think this bug is occurred because I tried to implement functions in UE4 way.

It is common sense that “Player” object has a camera as its children. (So called ECS, Entity component system)

But WebXR may needs different approach and/or solution. It is not built to be game engine like UE4.

Again , Thank you @trusktr

trusktr · October 18, 2021, 10:49am

In fact making the camera a child of any object is completely valid in Three.js, so this may be considered a bug because in this case making the camera a child of an object, while using a class like Water or Reflector, introduces the issue. Those are classes from the examples folder though, so it might not be high priority.

smarthug · December 7, 2021, 1:37pm

This PR fixes the problem.

github.com/mrdoob/three.js

WebXRManager: Fix XRCamera Local Space Behavior by Reverting #21964

mrdoob:dev ← zalo:revert-21964

opened 09:56PM - 18 Aug 21 UTC

zalo

+1 -4

This PR fixes a bug introduced in r130 by reverting how XR Camera calculations h…andle local space (by partially undoing #21964 (which appears to conflate `matrixWorld` and `matrix`)). This enables the XR camera to be parented to another moving object without odd behavior once again. If this isn't the correct fix; I'd be happy to test alternatives in the context of my application. Related issues: #21964 , https://github.com/google/model-viewer/pull/2464/files#r647866462

This bug has occured because WebXRManager’s updateCamera method is copying worldMatrix values into local position/rotation/scale variables.

Topic		Replies	Views
Finding the local camera.position in an XR session Questions camera , webxr	0	1104	February 10, 2022
WebXR Camera Height Questions webxr	1	611	August 18, 2020
WebXR camera position and rotation issue? Questions webxr	1	585	November 10, 2023
WebXR and threeJS setup layer Questions webxr	0	788	May 13, 2020
[Solved] Water2 has an opposing horizontal offsets in each eye in VR. Shader issue or browser projectionMatrix issue possibly? Jobs water , webxr	8	841	September 24, 2023

Related topics