Take a look here and here to read about some very similar situations, and possibly grab some useful ideas. Interestingly, (some) rotations affecting things are present there as well, the only difference is that the talk is about instanced geometries, but you can disregard that and just wrap you mind around the essential.
Basically, Three.js indeed has Object3D.frustumCulled to avoid drawing what’s not in camera view … but that only works effectively if the object is small enough to not cause problems on its own … and it’s not part of a merged geometry. In other words, like alluded above by me and others like @mjurczyk at his 2. point, merging geometries is fine if it’s done on a “volume” that more or less fits the camera view.
This idea is found in the links above as well, where it’s suggested to use “chunks” of merged geometries instead, with each chunk large enough to make a positive difference in the number of draw calls, but small enough to not negatively affect performance on its own. So, splitting your map into merged geometries of reasonable size (ideally, only what would fit in a standard camera view) could help. Frustum culling should work for the other merged geometries in the set which would not get rendered, while the merged geometry “in view” (or the one the player is positioned within) would not cause performance issues on its own.
It appears that you can test if objects are in the viewing frustum by using the object’s bounding sphere, but then, why do that if Three.js is doing it anyway in its frustum culling process. Another thing that should be kept in mind is that “standard” frustum culling doesn’t account for depth, that is and probably could be handled in other ways. So, don’t expect walls behind the wall that is seen by the camera to positively affect performance, because as far as I know, they are a different problem, with a different solution (and probably not entirely perfect either).