Facial Morphs on a SkinnedMesh Character

Questions
,
1

Hi all,

I’m using ThreeJS for the first time in a number of years and I’m struggling with the new version.

I have a skinned, rigged character that I’m importing from Blender with a number of named body animations as a glTF. I want to be able to control the facial expression - blinking, mouth phonemes etc. through JavaScript independently of the pre-canned animations.

I was planning on importing the facial expression morph targets in a separate glTF file. How do I then add a collection of morph targets to my SkinnedMesh that refers to these geometries?

I can find no help at all on morphAttributes and MorphTargetInfluences is a property of a Mesh object, but not a SkinnedMesh object as far as I can see.

any help or examples massively appreciated!

2

What is the reason for doing this? In general, you will have an easier workflow if you embed morph target animations into the same glTF file.

I suggest you study the following official three.js example that shows an animated robot based on skeletal animation. However, the facial expressions are implemented via morph targets.

https://threejs.org/examples/webgl_animation_skinning_morph

1 Like
3

@Mugen87

I’ve been using Mixamo.com to apply simple animations and create a rig for my character and using Blender to convert to glTF. I actually use 3DS Max for my morph targets but they were getting lost in the conversions. I’ve now figured out how to reapply them in Blender using Shape Keys and it’s working. Thanks for the help :+1:

2 Likes
4

Hey @GeorgeP
I am also working on adding Facial expressions on my 3d character with morph target animations.
I would love to know your progress and what things you achieved.
Can you guide me over this ?
I am trying to add lipsyncing and facial expressions to my 3d character model .

5

Hi @Anish_Kumar. I successfully created a 3D character with armature-driven body animations, facial expressions and dynamic lip sync, yes. Unfortunately the project is still ongoing and is commercially sensitive, so I can’t simply tell you how I achieved everything. I can offer some pointers though:

The characters are created such that each body part is a separate object. The face, eyebrows, teeth, mouth, tongue etc have morph targets for different expressions as well as visemes for lip syncing.

I use quite a small set of visemes - only 16 mouth shapes (based on the original Disney set) - but this is plenty for convincing lip sync. For dynamic audio you have to ensure that your provider gives the viseme data along with the audio (I use CereProc and Azure’s text-to-speech APIs because they both give the viseme data).

I do the actual face and mouth animations by creating a gsap timeline for each expression or speech utterance. In the case of lip sync, I actually attach the timeline to my audio object and I update its progress as the audio progresses. I found this gives much tighter synchronisation than simply playing the audio and timeline independently.

My digital assistants also use Azure to understand the user’s questions and responses and are now powered by OpenAI’s NLU, so they are walking, talking, conversational characters that are genuinely useful. The downside is that despite spending weeks on optimisation, my character + environment still weigh in at around 30MB, so I’m now starting to build a 2D fallback using Rive and I’ve been able to use an almost identical workflow for the dynamic lip sync based on Eli Sawicki’s excellent work: How exactly is Duolingo using Rive for their character animation?.

Here’s a sneak peek of an old version: https://www.youtube.com/watch?v=rAJW-KCNTiA

6

Hey @GeorgeP , Thanks for your response, you have done amazing work, loved to see your work.
A few doubts I want to ask,

  1. How you are handling the character bone animations (or hand gestures) like doing actions with hands while talking, how you are syncing it with the ongoing audio. Is this data is also coming form "CereProc’ and Azure or you are doing it statically. As I know Azure’s text-to-speech is only giving viseme data and facial expressions data.

  2. Are you adding facial expressions data as well or only lipsyncing you are doing. From Facial expressions I mean making happy, sad, angry and surprised etc faces while taking to emphasize the points while speaking.

  3. As I know CereProc and Azure both are paid services, right ? Do you know any free alternative for getting viseme data and facial expressions data as well ? Currently I am using rhubarb library which gives only data helpful for lipsyncing but I need to add facial expressions as well according to the context of the audio.

  4. Could you give me a example for how you are doing face and mouth animations with gsap (with breaking the confidentiality of you work), because I was trying to make an Animation clip with morpht target and viseme data values instead of a gsap animation for lipsyncing, but I didn’t succeeded.

Also can you expalin more about “I actually attach the timeline to my audio object and I update its progress as the audio progresses. I found this gives much tighter synchronisation than simply playing the audio and timeline independently.”.
Bcoz I was playing audio and lipsync animation independently.

Please forgive me😅, if I asked even minor and obvious thing, I am just a newbie in these things and seeking help.

Thanks In Advance!