So I found the problem, thanks to this thread. Seems like it was a depthBuffer precision issue.
I tried enabling logarithmicDepthBuffer, but that didn’t have any effect.
What solved it was setting the camera’s near & far clipping planes from default (0.1 - 2000) to 1 - 1000.
Solved the strange stepping issue and some other minor artifacts too.