Three.js face mesh mouth animation wrong when using landmarks / shader deformation

Questions
, , , , , , ,
1

Hello,

I am building a Jarvis-like AI face using Three.js.

System:

- Three.js rendering

- Face mesh with shader

- MediaPipe landmarks for face tracking

- Animation loop for face movement

- WebSocket events from backend

Everything works, but I have a problem with mouth animation.

Problem:

Lower lip / mouth movement looks wrong.

Mesh deformation is not correct when landmarks update.

Sometimes the lip goes inside / stretches incorrectly.

Face animation works for eyes and head, but mouth is not correct.

I think the issue is with:

- mesh deformation

- shader

- landmark mapping

- animation update

- morph / vertex movement

I am attaching screenshot.

Can someone guide how to fix mouth animation in Three.js face mesh?

This is part of a real-time AI face system, so animation must update every frame.

Thank you.

2

We need visual references! A screenshot or, better, a video.
There is a three.js example that you might be able to use.

3

Hi Phil,

Thank you for your response.

As you suggested, I’m sharing the UI files and visual references so the issue can be understood more clearly. google drive link : https://drive.google.com/file/d/1wkA833yP71Mrty89D1XeE45FFUCntXGn/view?usp=sharing

Project context:

I am building a real-time conversational AI avatar using Three.js. The face is currently driven by MediaPipe landmarks and shader deformation, with parameters like mouthOpen, phoneme shapes (A, E, O, U), and additional facial mechanics.

So far the system has progressed through about 43 integration phases including speech mechanics, phoneme mapping, emotion layers, gaze/blink behavior, breathing simulation, and AI-brain connection. The overall architecture works well, but a few rendering issues remain in the mouth area.

Main issues observed:

  1. Upper lip edge artifact
    Sometimes a light or white line appears on the upper lip edge during deformation. It looks like a shading or texture artifact when the mouth changes shape.

  2. Lip corner stretching
    During certain phoneme shapes such as A, O, or U, the lip corners occasionally stretch outward slightly and look unnatural.

  3. Teeth visibility (primary issue)
    When the mouth opens, instead of showing natural teeth or inner mouth detail, a dark gap appears between the lips. This makes speech look less realistic.

The system itself functions correctly in terms of speech logic and animation flow, but the visual deformation of the mouth region needs refinement.

I’m attaching the UI files and references so you can see the current setup. If you need a video reference of the animation in motion, I can share that as well.

Any guidance or suggestions on how to correct the deformation and improve the mouth rendering would be greatly appreciated.

Thank you again for taking a look.

Best regards,
Avishkar

4

No one’s going to download random files from some stranger online. Use GitHub—it’s standard practice.

Are you actually planning to code this yourself, or just experimenting?

Mapping realtime audio to realistic 3D facial animation is insanely complex and far from a solved problem. Even if you get the phonemes right—which is only an approximation—you’ll still miss all the subtle nuances that make speech look natural. That’s why high-end studios rely on extremely skilled animators.

You might get a rough approximation, but realistically, most people just create a baked animation and sync the audio to it. If your goal is realistic speech, you’re way out of your depth.