This is because of a change in r120 which is highlighted in the migration guide:
OrbitControls as well as other controls were moved to the Pointer Events API. In your case, you have to register your event listeners to pointerdown and pointerup events instead.
The reason behind this is that OrbitControls calls preventDefault() on Pointer Events in certain cases which automatically cancels the respective mouse events.