Hello everyone!
I’m working on a project where I’m using three-mesh-bvh for raycasting, and I’m implementing a sculpting functionality that allows users to add or remove material from a mesh surface using a brush. The brush size is adjustable for finer control.
To achieve a smooth surface, I use mergeVertices. I also use MeshStandardMaterial with vertexColors: true, as each vertex is assigned one of two colors. As a result, vertices with different color attributes are intentionally not merged, which leaves visible faces between regions of different colors. This behavior is expected and part of the design. The mesh itself has a complex, non-uniform shape with thin details.
For raycasting, I’m using a sphere with shapecast to control the radius of intersected vertices. However, I’ve encountered an issue in regions with thin geometry. When the brush size is increased, the shapecast sphere begins intersecting vertices from the opposite side of the geometry (as shown in the image below). This is problematic because I only want to affect the vertices on the front side. As a result, the calculation of the average normal (used for modifying vertex positions) becomes incorrect, causing the brush to remove material instead of adding it.
My first thought was to filter the vertices based on the dot product between ray.direction and the vertex normal. I implemented this approach within bvh.shapecast (see snippet below). While this partially solves the issue—it prevents material from being removed when it should be added—it introduces a new problem: the mesh begins to tear in some areas (as shown in the second image below).
Here’s the code snippet for my current approach:
bvh.shapecast({
intersectsBounds: (
box: Box3,
_isLeaf: boolean,
_score: number | undefined,
_depth: number,
nodeIndex: number
): ShapecastIntersection | boolean => {
accumulatedTraversedNodeIndices.add(nodeIndex)
const intersects = sphere.intersectsBox(box)
const { min, max } = box
if (intersects) {
for (let x = 0; x <= 1; x++) {
for (let y = 0; y <= 1; y++) {
for (let z = 0; z <= 1; z++) {
tempVec.set(
x === 0 ? min.x : max.x,
y === 0 ? min.y : max.y,
z === 0 ? min.z : max.z
)
if (!sphere.containsPoint(tempVec)) {
return INTERSECTED
}
}
}
}
return CONTAINED
}
return intersects ? INTERSECTED : NOT_INTERSECTED
},
intersectsTriangle: (
tri: ExtendedTriangle,
index: number,
contained: boolean
) => {
const triIndex = index
triangles.add(triIndex)
accumulatedTriangles.add(triIndex)
const i3 = 3 * index
const a = i3 + 0
const b = i3 + 1
const c = i3 + 2
const va = indexAttr.getX(a)
const vb = indexAttr.getX(b)
const vc = indexAttr.getX(c)
const na = new Vector3().fromBufferAttribute(normalAttr, va)
const nb = new Vector3().fromBufferAttribute(normalAttr, vb)
const nc = new Vector3().fromBufferAttribute(normalAttr, vc)
if (contained) {
if (na.dot(direction) < 0) {
indices.add(va)
accumulatedIndices.add(va)
}
if (nb.dot(direction) < 0) {
indices.add(vb)
accumulatedIndices.add(vb)
}
if (nc.dot(direction) < 0) {
indices.add(vc)
accumulatedIndices.add(vc)
}
} else {
if (sphere.containsPoint(tri.a) && na.dot(direction) < 0) {
indices.add(va)
accumulatedIndices.add(va)
}
if (sphere.containsPoint(tri.b) && nb.dot(direction) < 0) {
indices.add(vb)
accumulatedIndices.add(vb)
}
if (sphere.containsPoint(tri.c) && nc.dot(direction) < 0) {
indices.add(vc)
accumulatedIndices.add(vc)
}
}
return false
},
})
Screenshots:
Sample of thin geometry:
Resulting tearing issue:
Has anyone encountered a similar problem, or have suggestions on how to refine this approach to avoid tearing?
Thanks in advance for your help!
