Distance between facial landmarks changes with head rotation using MediaPipe Face Landmarks and Three.js

Questions
1

I’m using the @mediapipe/tasks-vision library to extract 3D face landmarks in a browser-based application and trying to measure the distance between two specific points: landmark 143 and landmark 372 (left and right eye corners).

Here’s how I’m calculating the distance using Three.js:

const distance = new THREE.Vector3(faces.faceLandmarks[0][143].x - faces.faceLandmarks[0][372].x,
  faces.faceLandmarks[0][143].y - faces.faceLandmarks[0][372].y,
  faces.faceLandmarks[0][143].z - faces.faceLandmarks[0][372].z
).length();

This seems to work fine when the user is looking straight at the camera. However, when the user rolls their head (tilts left or right), the calculated distance increases unnaturally, even though the actual distance between the two facial points hasn’t changed.

How can I get a consistent, rotation-invariant distance between two facial landmarks using MediaPipe’s 3D landmarks in a browser environment?

2

Vector3 has a distanceTo method, have you tried…

faces.faceLandmarks[0][143].distanceTo(faces.faceLandmarks[0][372])
3

Yes i have tried distanceTo method as well. it also gives the same result.
actually landmarks that i’m getting from mediapipe inconsistent, when i roll my head.

4

I don’t think you get that kind of accuracy from it. It’s just the models best guess based on segmentation.
The information you get from also has a fair bit of jitter generally… you can smooth the data by doing some filtering on the points/averaging them over time, so each sample only contributes a little bit to the current value.
Getting something that is smooth is tricky.

1 Like