I am starting from zero; I can’t find any resources which could explain me how to achieve automated lip animations. I know you can deform meshes over time with bones or morph targets. But I can’t find any resources explaining this for lip animation… do I have to have a target for each character (or phonetic characteristics) like here:
and then have the audio as text and then iterate through the array and animate it? how about facial expressions? and all the mixes of them?
The only resource in a book I found was from the old book “Three.js Cookbook”:
But that is just generic about morphs. Anyone who could help to implement this? I want to have a generic solution that I can adapt on different (same topology) models.
did you manage to have a look at the official face morph targets example? the setup demonstrates how you’d implement the resulting morph targets of a given model.
Thank you for the examples! Do I get that right, that my model need pre-processing some stuff in eg. Blender before I can access those morph targets and use morphTargetInfluences? It is not possible to define morph targets in code? Assuming I have a model of a head and the user selects the facial keypoints (or use tensorflow) and then let javascript do the rest?
I’m not sure how you’d procedurally create morph targets in javascript or three to be honest, it sounds like a next to impossible task, as @makc3d has said its likely best to create morph targets externally and then import one model with them all attached.
I did want to try to import the entire soft body and face rig at some stage that could allow full control over the facial skeleton, with control handles that limitedly combine expressions and movement, not sure of limitations but my guess is that weight maps may be a problem as well as other unforeseen incompatibilities…
But is it possible to apply the morph targets from model1 to another model2? Or should I have a look in the Blender API and when the user uploads a 3D model => try to copy the morph targets of an existing 3D model and apply it to the new model?
As I get it right: morph targets are just a stage with vertices and their positions? … means you could have different positions for each vertices and try to animate the vertices from Position1 to Position2… since you can iterate over the vertices and you have fixed topology and know which vertices are for the faces (so that you only iterate over those)… in theory it should be possible right?
Ah yes if you already know the positions that’s totally possible, you could just update the position buffer attribute, you can see the following example I recently experimented with that does this…
The pitfall would be that you’ll be loading all models that have different positions to transition between, morph targets are essentially compiled into one neat model that has each expression embedded…
Just adding here that pose morphs aren’t the only way for facial expressions, most games i know actually use bones or a combination of bones and morphs. With morphs you have more fine control while with bones much more flexibility in combination, also to make rotations.
Also to consider that morphs require a lot more memory, and bones are a bit harder to setup in a facial rig, also since influences need to stay at 4 per vertex.
I personally use morphs mainly baked for body customization, for facial animation bones or face tracking. If you use morphs it will be good for memory if the head is a separate mesh.
Not sure what you mean with topology, but you could remap vertex data onto different meshes. If the shape is the same remap spatially, if the topology/polygons are the exact same but in a different shape you can transfer it directly if the morphs are stored relative.
As long as the number of triangles and their placement is the same (and that is quite strong requirement, means the models must be made from the same template) you could do model 2 some target vertex = model 1 some target vertex - model 1 vertex + model 2 vertex, but even then it could look shitty. Better just make an artist do all those targets by hand.
i haven’t seen one as of yet, i did quite of bit of thinking into how this could be acheived at the time of creating the example but there was so much to take on board, it essentially breaks down into phonetic approximation between vowels, constenants and syllables in combinations of the entire alphabet, ex-human made some astonishing progress on this front, especially when chaining libraries with langchain, see this WolframAlpha example… i’m really not sure how this could or would translate into morph target interpolation
I think one of the most simple interpolations that cover all mouth shapes is the Aardman model
From the example I’ve shared above you’d essentially create these mouth shapes and then somehow detect when combinations of such arise in any text to speech algorithm, moving accordingly towards the relative morph target of each detected phonetic shape.
