For anyone happening upon this later on, you can find a working example here :
Solution : three-fiber-zoom-to-object - camera-controls solution - final - CodeSandbox
This is just a slight change on @drcmda 's implementation of camera-controls
with normal lerping to move the camera. It’s not perfect (for one, the transition time in camera controls doesn’t seem to be editable, so there’s a weird swing-around thing that happens, when you’re zooming back out) but it definitely solves the problem. (Many thanks to @looeee and @Lawrence3DPK for additional help.)