I have dissected your character controller and applied it to cannon.js creating a localized collider geometries. I never thought of this would be the solution to the large scale environment now the physics is localized to the player where the player interact with environment ray casting only the mash close close to the player.
Thanks so much @hh-hang this really revolutionized three.js virtual experience.