I thought about it a bit, you could trace paths using Djikstra’s from every polygon to every other polygon, and find looping paths. If there exists a looping path who’s combined curvature is positive - relative to triangle normal - you found a path that forms a hole. When i say “positive curvature”, I mean that the next triangle in the path has its normal pointing inwards toward the previous triangle’s normal.
The performance of such an algorithm would be O(n^3), where n is number of triangles. Which is to say - it would be pretty bad, but it would work. And it would find all loops that contribute to topological holes.