Moving around a hollow earth with a dynamic up vector, eventually pulls the player towards the north pole

Hello all, I’m creating a hollow earth kind of world where the player moves along the inside of a sphere. I’m trying to figure out how to get the movement/camera working correctly for this. My experience with rotations is pretty limited, but I’ve tried a couple solutions that almost seem to work but they always pull the player towards a certain point on the sphere and then the player rotates around that point, which seems like a gimble lock issue. Here is an example using quaternions (which I assumed would prevent gimble lock). If you walk forward for a bit you’ll see the north pole (marked by a red box) and if you try moving away from it then it keeps pulling you in.

I’ve been trying different solutions for weeks now so if anyone knows how to correctly calculate rotation for this you would have my eternal gratitude!