Combining threeJS gizmo in R3F - jumping camera issue


I am working on a React Three Fiber Project where I want to use gizmo (custom viewHelper class) from threeJS.

This is threeJS sandbox with gizmo I want to use in R3F project:

R3F with embedded above gizmo:

Almost everything is fine, but the when manipulating gizmo sphere (the transparent one) it jumps in R3F. Also when rotating it ( so the entire scene too) there is a jump between z and -z along Y axis.

My investigation:

Of course the issue starts in the onPointerDown method where we can see camera.position jumping if logging it in console

I see changing the behaviour of the gizmo if controls are not being set with the method setControls
then the gizmo jumps along X axis to -Y but Z doesn’t jump anymore.
if runed off this line there is no jumping camera effect, but of course then the orbitControls doesn’t change the position of gizmo

    updateOrientation(fromCamera = true) {
      if (fromCamera) {
        //turning it off solves the issue
        // this.quaternion.copy(;

I saw React Drei Gizmo, but they are less interactive