I came across this same issue as you ShawnWhy, and after a bit of frustrating debuggin, I managed to solve the issue. Just in case someone else gets stuck on the same thing… If you have an object in layer 3 and want it to render in VR, this works:
const VRCamera = renderer.xr.getCamera();
VRCamera.layers.enable( 3 );
for (const camera of VRCamera.cameras){camera.layers.enable( 3 );}