This is normal situation and it can be quite shocking when you experience it for the first time. The problem is this: Orbit controls maintains consistent up direction along the Y-axis. Your code sets the camera at (512,800,-256) and turns it towards (512,0,-256). This means that the Y-axis is destroyed, because it coincides with the view direction.
The visible rotation appears “random” due to the turbulence of errors.
The easiest way to resolve this is by introducing a small change in the camera position. Change line 27 from:
camera.position.set(x, 800, y);
camera.position.set(x, 800, y+1);
The shift is so small, that users will not notice it, but it is big enough to overcome fluctuations of errors.
However, the best solution would be to change the orientation of the map, so that the view direction is along the Z axis, not the Y axis.
Edit: Y-coordinate is the middle one; Z-coordinate is the last one. So, the variable called y is used as Z-coordinate.
I’m not quite sure what effect you are trying to achieve. Here is a demo in pure JS, because I’m not familiar with React. The demo shows a map of the Earth. It can be zoomed, panned and rotated, but not flipped. The navigation is with OrbitControls. I hope this helps you a bit.