you can change the camera position and orbit controls target any time you wish.
if you change camera rotation outside of the OrbitControls methods, then call OrbitControls update so that the OrbitControls uses the new camera matrix.
See this below example where I use the perspective camera and OrbitControls together and change the properties several different ways. I.e., buttons and double click on the scene to lerp target and position while still allowing OrbitControls to control.
