After some trial and error it appears that my setting of renderer.outputEncoding = sRGBEncoding; to get the GLTF models to look correct, has no effect in VR so they go back to being too dark again.
Is there a correct solution here or should I avoid using that and fake it with an ambient light or tone mapping exposure or something?