Best approach for real-time facial morphing (nose, chin, lips) in Three.js

Questions
, , , , , , ,
1

Hi everyone,

I’m working on a project where we receive a 3D scan of a patient’s face (from a mobile app).
We want to build a web-based tool where users can interactively change facial features — for example:

  • Increase/decrease nose size (width/length)

  • Change chin length/position

  • Adjust lips volume

The goal is to give a real-time preview before surgery.

Some details:

  • The face model will be loaded in Three.js (format can be GLTF/OBJ, we have control over export).

  • We need smooth deformations (not just moving vertices manually) and must keep the texture realistic.

  • Ideally, we’d like sliders for each parameter.

My questions:

  1. What is the best approach in Three.js for this kind of morphing?

    • Morph targets / blend shapes?

    • SkinnedMesh with bones?

    • Custom vertex shader deformation?

  2. If we use morph targets, how should we prepare the models? (e.g. need multiple morphed versions of the face with different parameters?)

  3. Any recommended libraries, tools, or examples for real-time face morphing in the browser?

Thanks for any advice or example code!

2

Check this one might be helpful hand-tracking-threejs

3

Probably mesh skinning. But it’s going to be a tricky prospect.. you’ll have to find a way to automatically rig the scanned mesh onto your face rig.

There is a new opensource autorigger for bodies that may serve as inspiration… but this is a pretty highly technical task:

1 Like
4

I think for 8thwall we have to pay

5

Do we need rig for face morphing?I thought rig related to animation.What do you think about using morph target?

6

maybe use media pipe face landmarker to get the face mesh https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker

and since the vertices will be in the same order , you could try manipulating the nose vertices to make it wider longer etc

1 Like