I don’t think it drops the camera position. It would be just appearing as though it has dropped the position because it is looking at the origin.
Doing console.log(camera.position.x) confirms it.
Also, always do controls.update() after transforming the camera.