ok, so I have researched this for quite some time and could not find a working solution. I’m starting to think this is a bug with the OrbitControls. So when we use the OrbitControls, the controls.update() call is not performed at all on each animation loop. That will ensure we can dynamically rotate our camera.
The problem is when we activate the controls by dragging our mouse, the camera’s rotation is reset back to the normal horizontal plane. How can we prevent the controls from doing that rotation reset?
I’d rather not go into ThreeJS’s OrbitControls module and tweak it and hope there is a way to achieve this.
That’s exactly the job of OrbitControls though - it automatically updates the camera to always look at the target, overriding on each frame all other changes you manually apply to the camera in code.
If you use controls - let them control the camera, and do not modify it yourself - instead, use .target of the controls to modify the direction in which controls should point the camera.
hmm, yes, setting target only allow me to set the “focused” / origin point of the controls. My camera has rotated and reverts back to some initial controls angle if I rotate/pan around. I would like the OrbitControls to please leave my camera’s rotation alone. I have started to go into the OrbitControls module just now and may have to do something funky in there…
See the demo here:
So in the OrbitControls module, this is where the magic happens when we rotate the controls. Time to see if that can be tweaked to spare my camera’s rotation.
function handleMouseMoveRotate( event ) {
rotateEnd.set( event.clientX, event.clientY );
rotateDelta.subVectors( rotateEnd, rotateStart ).multiplyScalar( scope.rotateSpeed );
const element = scope.domElement;
rotateLeft( 2 * Math.PI * rotateDelta.x / element.clientHeight ); // yes, height
rotateUp( 2 * Math.PI * rotateDelta.y / element.clientHeight );
rotateStart.copy( rotateEnd );
scope.update();
}
Not sure if code modifications will be necessary - OrbitControls.enableRotate exists - but disabling rotation kinda kills the point of using the OrbitControls 
it leaves only the pan / dolly there, which can be done quite a bit easier and simpler just using mouse events.
Rotation will have to remain enabled. The OrbitControls will need to learn how to rotate up/down and side-to-side when the camera is rotated. So I sense some new math calculation will be required to tell the update() function to take into consideration the rotation. Seems like the code tweak will need to go in that direction…
Looks like the method to look more closely at is the .lookAt()…
If you rotate the camera, it will no longer look at the target - making orbit controls no longer orbit the target, making them no longer orbit controls 
(So as above - instead of rotating camera directly, consider “rotating” the target position around the camera’s up-axis. That’ll allow the orbit controls to keep following the target properly, and let you rotate the camera at will in a way.)
heheh, the targets or Object3Ds will have their own rotation/quaternion and the camera (“us”) is obligated to rotate to match their rotation when facing them.
Here is the code for the lookAt method at I am highly suspicious about:
if ( this.isCamera || this.isLight ) {
_m1$1.lookAt( _position$3, _target, this.up );
}
Especially the this.up portion… I suspect is the “issue” here. If I can somehow modify/rotate the up value somehow, that could do the trick… What is “up” really when you are in space is the question. lookAt() should really not make that parameter as required.
After you change the camera… do. Controls.target.set(0,0,-5).applyQuaternion(camera.quaternion).add(camera.position)
Right at the beginning, I did this to do a simple camera rotation on the z-axis:
camera.rotateZ(0.5432);
controls.target.set(0, 0, -5).applyQuaternion(camera.quaternion).add(camera.position);
The moment I rotate the controls, it jumps back to the up position.
After looking into the ThreeJS code, I found that the rotation camera back to the “up” position is done with this code:
_m1$1.lookAt(_position$3, _target, this.up);
this.quaternion.setFromRotationMatrix(_m1$1);
If somehow we could adjust the value of the Matrix4 m1$1 to rotate on the z-axis to match the z-axis rotation of our target Object3D, I think that would work. Now, what in the world is a Matrix4…? 
wahh, _m1$1.makeRotationZ(0.1234) did not do the trick. This lookAt() with the fixed “up” value is really messing things here I suspect. 
_m1$1.lookAt(_position$3, _target, this.up);
Interesting. The TrackBallControls updates the up value of the camera as we rotate it.
