[Resolved] Surface Boolean Without CSG — The Full Solution
Follow-up to my original post: Surface Boolean Without CSG — Closing the Seam After Split-and-Classify
Thanks to @PavelBoytchev and @hofk for the helpful suggestions — the edge-stitching demo and Hartmann triangulation reference got me going towards the light. I wanted to post the full-resolution version since this is a problem I couldn’t find a complete solution to online, and others working with open triangle meshes (terrain, geological surfaces, DTMs) may hit the same wall.
I abandoned the "split both surfaces independently, then try to stitch the seam afterwards" approach entirely. *
*
**The solution was to classify triangles geometrically (ray-casting + flood fill) rather than splitting them at the seam first, then only split the straddling triangles using Constrained Delaunay Triangulation with shared Steiner points.
This eliminates the seam mismatch problem at its root.**
Image of Extruded Solid and Terrain Mesh from OBJ 
︎
Image of Selection Panel ︎
Image of Split Selection Panel ︎
Image of Selected Splits to create a Solid ︎
Image of Completed Solid and Statistics ︎
The Problem (recap)
Two overlapping open triangle meshes (e.g. terrain surface + pit design), need to be split along their intersection and selectively merged — a Maptek Vulcan “TRIBOOL” style operation. Not solid CSG — these are open DTM surfaces that can’t be closed into watertight solids.
My original approach split both surfaces along the intersection segments independently, which produced near-coincident but non-identical vertices along the seam. No amount of post-hoc stitching, welding, or capping could reliably close the resulting gaps.
What Actually Works — The Algorithm
The working solution has 9 steps. Here’s the pipeline:
Step 1–2: Intersection Detection
Extract triangle soups from both surfaces, then run Möller tri-tri intersection on all A×B triangle pairs, accelerated with a spatial grid. Each intersection produces a segment {p0, p1} tagged with source triangle indices {idxA, idxB} on both surfaces. For a typical terrain (27k tris) vs pit shell (40 tris), this produces ~230 intersection segments in under a second.
Step 3: Build Crossed-Triangle Maps
From the tagged segments, build a map per surface: triIndex → [segments that cross this triangle]. Any triangle appearing in this map is “crossed” — it straddles the intersection boundary. All other triangles are cleanly on one side.
Step 4: Spatial Grids for Classification
Build XY spatial grids over both surfaces for fast ray-casting lookups (used in the next step). Cell size is ~2× the average edge length.
Step 5: Flood-Fill Classification (the key insight)
This is where the old approach went wrong. Instead of trying to classify by Z-height interpolation or signed distance, I use connected-component flood fill with ray-cast seeding:
-
Build edge adjacency for all non-crossed triangles (crossed triangles are excluded — they form the boundary)
-
BFS flood fill finds connected components — regions of triangles that share edges but are separated by crossed triangles
-
Each component gets classified by one seed triangle: cast a +Z ray from the seed’s centroid through the other surface’s triangles, count crossings. Odd = inside, even = outside.
-
The seed’s classification propagates to the entire component.
This works because crossed triangles naturally break the adjacency — triangles on opposite sides of the intersection can’t share edges with non-crossed triangles that span the boundary. The flood fill respects this natural partition.
// Classify seed via ray casting against other surface
var seedTri = tris[seed];
var cx = (seedTri.v0.x + seedTri.v1.x + seedTri.v2.x) / 3;
var cy = (seedTri.v0.y + seedTri.v1.y + seedTri.v2.y) / 3;
var cz = (seedTri.v0.z + seedTri.v1.z + seedTri.v2.z) / 3;
var seedClass = classifyPointByRayCast(
{ x: cx, y: cy, z: cz },
otherTris, otherGrid, otherCellSize
);
// BFS: propagate to entire connected component
var queue = [seed];
while (head < queue.length) {
var curr = queue[head++];
for (var ni = 0; ni < neighbors[curr].length; ni++) {
var nb = neighbors[curr][ni];
if (!visited[nb]) {
visited[nb] = 1;
result[nb] = seedClass;
queue.push(nb);
}
}
}
The ray-casting PIP (point-in-polygon) works in all orientations — closed pit shells, open terrain, vertical walls. Vertical faces are automatically skipped (degenerate XY projection), so the test gracefully handles steep geometry.
Step 6: Split Straddling Triangles (Constrained Delaunay)
Non-crossed triangles go directly to their inside/outside group. Crossed triangles need subdivision — but instead of splitting along the intersection segments and hoping vertices match, I re-triangulate each crossed triangle with all intersection segment endpoints as Steiner points:
-
Build a local 2D coordinate frame on the triangle’s plane (critical for steep/vertical triangles — using world XY causes incorrect splits)
-
Collect all segment endpoints that fall inside the triangle (barycentric validation)
-
Run Delaunator on the original 3 vertices + Steiner points
-
Constrain the intersection segment edges using Constrainautor ( THIS LIBRARY HAS SAVED ME SO MANY TIMES boundary edges are NOT constrained — they’re the convex hull already, and constraining them skips intermediate Steiner points on boundary edges)
-
Filter sub-triangles: centroid must be inside the original triangle (barycentric test), area must exceed a minimum threshold
-
Classify each sub-triangle independently via ray casting
This is the crucial difference from my old approach:
- Both surfaces’ straddling triangles are classified using the same geometric test (ray-cast against the other surface), not by trying to align split vertices.
- There’s no seam to close because the classification is position-based, not topology-based.
Step 7: Seam Vertex Deduplication
After splitting, near-coincident vertices along the seam are merged using a 3D spatial grid (tolerance 0.1mm). This uses a 3×3×3 neighbourhood search for O(n) performance. Degenerate triangles (where vertices collapsed to the same object) are removed.
Step 8: Normal Propagation
Enforce consistent winding order across the result mesh. For manifold meshes: BFS from a seed triangle, checking shared-edge direction — adjacent triangles should traverse their shared edge in opposite directions. For non-manifold meshes: fall back to per-triangle Z-up normals.
Step 9: Interactive Merge
The user sees 4 colour-coded groups (A-inside, A-outside, B-inside, B-outside), toggles which to keep, and clicks Apply. The merge pipeline handles vertex welding, degenerate removal, optional boundary stitching/capping, and stores the result as a new surface.
The Tool, once tested on more diverse surfaces, might get a list form for use with multi-meshes
The Merge Pipeline (Closing the Surface)
To get a watertight result, the merge includes an optional “stitch” mode with this cascade:
-
Weld vertices — spatial grid O(n) merge within user-specified tolerance
-
Remove degenerates/slivers — area check + minimum-altitude/max-edge ratio test
-
Clean crossing triangles — remove over-shared edges by keeping the 2 largest triangles per edge
-
Stitch by proximity — for each boundary edge, find the nearest boundary edge within tolerance where BOTH endpoints match, connect with a quad (2 triangles)
-
Sequential boundary capping — extract boundary loops, cap one at a time with Constrained Delaunay, re-weld and clean non-manifold edges between each cap pass. This avoids the double-cap problem where capping all loops at once creates overlapping triangles.
-
Safety net: force-close — operates on the indexed mesh (integer point indices, zero float precision issues), iteratively finds the nearest valid point for each remaining boundary edge
The key to reliable capping was sequential loop processing — cap one loop, clean up non-manifold artifacts, then cap the next. All-at-once capping was the source of most of my pain in the original post.
NOTE ABOUT CSG: For open surfaces, CSG produces unreliable results (normals are undefined for open meshes), which is why the Surface Boolean approach above is necessary.
What Didn’t Work (So You Don’t Have To Try)
For others attempting this, here’s what I tried and abandoned:
-
CDT + flood-fill on the split mesh: Bridge triangles leak across the seam because CDT constraints can fail on edge conflicts, and the flood fill crosses cut edges.
-
Z-interpolation classification: Comparing the centroid Z against the other surface via barycentric interpolation. Fails for vertical walls, overhangs, and any geometry that isn’t a simple heightfield.
-
Connected-component detection with cut-edge exclusion: Recording which edges were created by splitting and excluding them from adjacency. Works for terrain but fragments the pit shell into hundreds of single-triangle components.
-
Two-pass component refinement: Sub-splitting base components with cut edges, accepting only if the result isn’t fragmented. Theoretically sound, but the heuristics for “fragmented vs legitimate split” were unreliable.
-
Post-hoc stitching without classification-first: The original approach from my first post — every technique (weld, stitch, cap, curtain, force-close) was fighting the fundamental problem that the two surfaces had different vertices along the seam.
Libraries Used
-
Three.js r183
-
Delaunator — unconstrained Delaunay triangulation
-
@kninnug/constrainautor — constrained Delaunay on top of Delaunator
-
THREE-CSGMesh — solid CSG operations (for closed meshes)
-
MeshLine — fat-line rendering for intersection polyline preview
Live Demo & Source
Update — Fixed! (v0.2.0)
The angled wall classification issue is now resolved.
1. Multi-Axis Jittered Ray Casting
The original code cast a single ray per axis. If that ray landed exactly on a triangle edge (common with axis-aligned geometry), both adjacent triangles counted the hit → even count → wrongly classified as “outside”.
Fix:
Cast 3 deterministic jittered rays per axis (~0.00005 offset in the projection plane), majority vote
per axis. Each axis returns a 3-state result:
0 = no hits (no vote — ray missed the mesh entirely)1 = inside (2+ of 3 jittered rays had odd hit count)2 = outside (2+ of 3 jittered rays had even hit count).
Then the multi-axis vote across Z, X, Y: 2+ axes say “inside” → inside. Handles any wall angle 0–90° without thresholds.
2. Free-Vertex Topology Inheritance (the critical one - Vertex-Adjacency Sub-Triangle Classification)
After CDT splitting at intersection edges, each sub-triangle needs classification. The old approach ray-cast the centroid — but centroids of sub-triangles sit right next to the intersection boundary where ray casting is least reliable.
Fix: Build a vertex→classification map from non-crossed triangles (flood-fill results).
For each sub-triangle, find a vertex that is NOT a Steiner point (not on the intersection line),
look it up in the map, and inherit that classification. Pure topology — no ray casting at the
boundary. Only falls back to ray casting when no adjacent non-crossed triangle exists.
The fix uses pure topology instead of geometry. Here’s how it works:
- Build a vertex→class map from all non-crossed triangles (these already have correct flood-fill classifications). Every vertex of every non-crossed triangle gets mapped to its triangle’s inside/outside class.
- Collect Steiner keys — the intersection segment endpoints that were inserted by CDT. These vertices sit ON the intersection line and belong to both sides, so they can’t tell us anything.
- For each sub-triangle, find the “free” vertex — the one that is NOT a Steiner point (not on the
intersection line). This vertex was an original mesh vertex that also belongs to a non-crossed neighbour triangle.
- Look up the free vertex in the map → inherit that classification directly. No ray casting, no geometry queries. Pure topological adjacency.
- Fallback: Only if ALL vertices of a sub-triangle are Steiner points (rare — happens when a tiny triangle sits entirely on the intersection line) do we fall back to multi-axis ray casting.
3. No More Angle Thresholds
Removed surfaceNeedsMultiAxis — always test all 3 axes. The jitter + majority vote is cheap enough that there’s no reason to gate it.
This is the key insight: the flood-fill already classified the entire mesh correctly on both sides of the intersection. The CDT split just subdivided triangles at the boundary. Each sub-triangle shares at least one vertex with a non-crossed triangle on the correct side — so we inherit instead of re-classifying.
Result: Cube vs cube union produces a clean merged shell. Steep pit shells (80°+ walls) vs terrain work correctly. No more jagged edges at intersection boundaries.
Library updated:
https://www.npmjs.com/package/trimesh-boolean ·
trimesh-boolean — Live Demo
