I haven’t come back to this project yet, but I’ve been thinking about solutions. Could I only put the camera in the group when entering VR mode? Or … could I have two cameras, one for 2D and one for 3D and switch between them?
Yeah use another camera until entering VR, or a potential sudo idea would be to have a ghost secondary camera with orbit controls on it and apply this ghost camera matrix to the parent group containing your primary camera (maybe?)
I think that you should not try to use “OrbitControls” when entering VR, and instead implement an alternate method of manipulating the scene that makes use of the VR controllers. GoogleEarth VR provides a good example of how a scene can be navigated with your controllers.