started · updated
Researchers develop new AI systems for lip reading and natural character movement
Researchers have introduced two distinct AI advancements aimed at improving how machines interpret and replicate human communication.
At the Korea Advanced Institute of Science and Technology (KAIST), scientists developed DLLM-VSR, an AI system designed to read lip movements and convert them into text. To overcome the difficulty of distinguishing visually similar mouth movements, the system utilizes a diffusion large language model. Instead of processing words sequentially, the model identifies high-confidence words first and uses them as context to fill in the remaining sentence. The technology achieved a 19.4% word error rate on standard benchmarks.
Separately, researchers from Tohoku University and the University of Hong Kong developed DynaConTalk, a system that generates three-dimensional body movements to match spoken words. The technology aims to make digital characters appear more lifelike by coordinating hand gestures, facial expressions, and posture changes with speech patterns. By separating slow and fast movements and analyzing speech rhythm and meaning, the system produces more natural transitions than previous methods.
Entities
DLLM-VSR · DynaConTalk · Korea Advanced Institute of Science and Technology · Tohoku University · University of Hong Kong