How to filter vertices of merged geometry during raycasting?

Hello everyone!

I’m working on a project where I’m using three-mesh-bvh for raycasting, and I’m implementing a sculpting functionality that allows users to add or remove material from a mesh surface using a brush. The brush size is adjustable for finer control.

To achieve a smooth surface, I use mergeVertices. I also use MeshStandardMaterial with vertexColors: true, as each vertex is assigned one of two colors. As a result, vertices with different color attributes are intentionally not merged, which leaves visible faces between regions of different colors. This behavior is expected and part of the design. The mesh itself has a complex, non-uniform shape with thin details.

For raycasting, I’m using a sphere with shapecast to control the radius of intersected vertices. However, I’ve encountered an issue in regions with thin geometry. When the brush size is increased, the shapecast sphere begins intersecting vertices from the opposite side of the geometry (as shown in the image below). This is problematic because I only want to affect the vertices on the front side. As a result, the calculation of the average normal (used for modifying vertex positions) becomes incorrect, causing the brush to remove material instead of adding it.

My first thought was to filter the vertices based on the dot product between ray.direction and the vertex normal. I implemented this approach within bvh.shapecast (see snippet below). While this partially solves the issue—it prevents material from being removed when it should be added—it introduces a new problem: the mesh begins to tear in some areas (as shown in the second image below).

Here’s the code snippet for my current approach:


bvh.shapecast({
  intersectsBounds: (
    box: Box3,
    _isLeaf: boolean,
    _score: number | undefined,
    _depth: number,
    nodeIndex: number
  ): ShapecastIntersection | boolean => {
    accumulatedTraversedNodeIndices.add(nodeIndex)

    const intersects = sphere.intersectsBox(box)
    const { min, max } = box
    if (intersects) {
      for (let x = 0; x <= 1; x++) {
        for (let y = 0; y <= 1; y++) {
          for (let z = 0; z <= 1; z++) {
            tempVec.set(
              x === 0 ? min.x : max.x,
              y === 0 ? min.y : max.y,
              z === 0 ? min.z : max.z
            )
            if (!sphere.containsPoint(tempVec)) {
              return INTERSECTED
            }
          }
        }
      }

      return CONTAINED
    }

    return intersects ? INTERSECTED : NOT_INTERSECTED
  },
  intersectsTriangle: (
    tri: ExtendedTriangle,
    index: number,
    contained: boolean
  ) => {
    const triIndex = index
    triangles.add(triIndex)
    accumulatedTriangles.add(triIndex)

    const i3 = 3 * index
    const a = i3 + 0
    const b = i3 + 1
    const c = i3 + 2
    const va = indexAttr.getX(a)
    const vb = indexAttr.getX(b)
    const vc = indexAttr.getX(c)

    const na = new Vector3().fromBufferAttribute(normalAttr, va)
    const nb = new Vector3().fromBufferAttribute(normalAttr, vb)
    const nc = new Vector3().fromBufferAttribute(normalAttr, vc)


    if (contained) {
      if (na.dot(direction) < 0) {
        indices.add(va)
        accumulatedIndices.add(va)
      }

      if (nb.dot(direction) < 0) {
        indices.add(vb)
        accumulatedIndices.add(vb)
      }

      if (nc.dot(direction) < 0) {
        indices.add(vc)
        accumulatedIndices.add(vc)
      }

    } else {
      if (sphere.containsPoint(tri.a) && na.dot(direction) < 0) {
        indices.add(va)
        accumulatedIndices.add(va)
      }

      if (sphere.containsPoint(tri.b) && nb.dot(direction) < 0) {
        indices.add(vb)
        accumulatedIndices.add(vb)
      }

      if (sphere.containsPoint(tri.c) && nc.dot(direction) < 0) {
        indices.add(vc)
        accumulatedIndices.add(vc)
      }
    }

    return false
  },
})


Screenshots:

Sample of thin geometry:

Resulting tearing issue:
Screenshot 2024-10-02 at 08.13.27

Has anyone encountered a similar problem, or have suggestions on how to refine this approach to avoid tearing?

Thanks in advance for your help!

If you have a “seam” in your vertices due to using different vertex attributes then you’ll get a separate in the edge if you only move one of the vertices, so you have to move any other vertices that are at that same position. There’s nothing in three.js that does this for you, though. You’ll have to create a data structure to keep track of any related vertices and move all the associated ones if you move one of them.

1 Like

Can you do the brush influence computation in a vertex shader instead of iterating/colliding the mesh?

Unfortunately, the only way I know to implement brush sculpting is through raycasting. I’m not sure how to manage it in a shader or how to pass variables from the application to the shader in a synchronized way. This sculpting feature needs to select a group of vertices and apply changes based on conditions stored in a separate data structure. I also need a way to update this data structure during editing.

You can store your vertices in a float rendertarget. you can pass in the brush info in a uniform. you read vertices from one rendertarget and output the modfied vertices to a second rendertarget, then swap the 2 targets. And you also can modify the material shader via onBeforeCompile to grab its vertices out of the texture instead of from the real vertex stream.

You can apply similar techniques to implement texture painting.

https://manthrax.github.io/monkeypaint/index.html?1

This is all just for editing/warping vertices though. If you’re talking about using CSG to actually remove/chop geometry up, then yeah… mesh-bvh-csg is probably a good bet.

three-mesh-bvh also has a whole sculpting package if you haven’t seen it yet:

https://gkjohnson.github.io/three-mesh-bvh/example/bundle/sculpt.html

1 Like

Thank you so much for the tip! I’ll definitely check it out :blush:

Regarding https://gkjohnson.github.io/three-mesh-bvh/example/bundle/sculpt.html my entire implementation is based on it (I referenced it). In the sample, a sphere is used, so there’s no issue with intersecting vertices whose normals point in the same direction as the raycasting ray. However, in my case, I am working with a more complex shape that has thin details. As a result, I need to account for an edge case where, due to a large brush radius, I unintentionally select vertices from the opposite side.

1 Like

Right. I think there you are on the right track with comparing the vertex normal facing with the cursor ray direction.