Strange behaviour when skipping react render cycle R3F

Hello guys I hope someone can help me understand whats going on and if what I am doing is some kind of anti-pattern. I am trying stop react from rerendering the AnimatedBox component and only update the values in useFrame.
In the first picture state is a simple js object that has its properties updated by a socket, its not reactive so it does not trigger any re-render of the component except the first one.
In the second picture the state is zustand state which triggers AnimatedBox rerender on update.

Both approaches move and rotate the cube. The problem is that the one with the simple js object is moving and rotating slower visually, but its receiving the same data. They are doing the same thing the only difference is that one rerenders on update and the other one doesnt.. I dont understand why is this making any difference in the scene