Try this: before switching to OrbitControls dispose it, recreate it and switch to the new instance. This is not an elegant approach, but if this works (I’ve never tried it), you will have some basic solution until you find a better solution.
There was a long discussion here about the problems of switching from one controls to another: