How to make the model mouth talking animation or lipSyncing?

Questions
, , , , ,
1

Hey I am trying to make the model mouth talking or lipSyncing. I have a audio file, and a model with keyshapes or morph target animation. I want that when I paly audio then the model morph target animaiton should play so that it look like model is speaking. I am following the below youtube tutroial of lipsyncing in react-three-fibre, but i want to implement it in Three.js.

Youtube Tutorial: https://www.youtube.com/watch?v=egQFAeu6Ihw
Article: https://www.wawasensei.dev/tuto/react-three-fiber-tutorial-lip-sync

I am facing problem to run the morphTarget animation insdie update method, the animaion is lagging too much

playLipSyncAnimation() {

		if (this.audio.isPlaying) {
			const context = this.audio.context; // Get the audio context from the audio object
			const currentAudioTime = context.currentTime; // Get the current time of the audio playback
		

			Object.keys(LIP_POSE).forEach((value) => {
				this.head.morphTargetInfluences[
					this.head.morphTargetDictionary[value]
				] = 0;
				this.teeth.morphTargetInfluences[
					this.teeth.morphTargetDictionary[value]
				] = 0;
			});

				
			
			for(let i=0; i<this.lipSyncJson.mouthCues.length; i++){
				const mouthCue = this.lipSyncJson.mouthCues[i];
				if( currentAudioTime >= mouthCue.start && currentAudioTime <= mouthCue.end) {
					console.log("Yes")
					this.head.morphTargetInfluences[
						this.head.morphTargetDictionary[LIP_POSE[mouthCue.value]]
					] = 1;
					this.teeth.morphTargetInfluences[
						this.teeth.morphTargetDictionary[LIP_POSE[mouthCue.value]]
					] = 1;
				}
			}

		}
	}

	update() {
		
		if(this.audio){
			if (this.controls.play) this.audio.play();
			else this.audio.stop();
		}
		
		if (this.allResourcesLoaded) {
			this.playLipSyncAnimation();
		}
	}

Inside tutorial also, he is playing the animation inside update function, so why its not working in my case.
My Code: GitHub - Anish-Someashwara/Threejs-Mouth-LipSync

My Output:

@donmccurdy @Mugen87 Seeking help!