Move camera with quaternions and tween.js

What if I use OrbitControls, where the camera is always looking at the target, but I want to use a button that also starts a movement above the target which would also trigger a gimble lock? Would I need to disable the controls, tween the quaternion, tween the position and then turn it back on?
Sadly when setting orbitControls.enable = false I still can control it which seems to be a bug.

At the moment I have exactly that issue Animation-with-tween-js/18549