Let’s be honest, nothing pulls you out of a game faster than a character whose lip movements have absolutely nothing to do with what they’re actually saying. It is a persistent headache, so much so that some developers simply give up on lip-syncing entirely—a problem that becomes even more glaring during localisation. In this brave new world of generative AI, however, we’ve seen a flurry of tech demos promising to fix our facial animation woes. A recent talk at the NEXUS Co., Ltd. Developer Conference (NDC) dived deep into how this is actually being put to work.

On Wednesday the 17th, at NDC 26, Han-yong Jang, Director of NC AI’s Physical AI Lab, took the stage to discuss 'From Speech to Expression: Streamlining Facial Animation with Generative AI.' Jang argued that as the baseline for in-game graphics and body animation continues to rise, the quality of facial animation has become the next critical frontier for developers.

Historically, we’ve relied on motion capture, but the fidelity often lags behind body movement, forcing artists to step in and fix things by hand. While NVIDIA’s AI solutions once offered a glimmer of hope, they struggled with the specific quirks of game audio—like intense acting or heavy reverberation—leading to distracting lip tremors. Epic Games had a go at it too; being an engine developer, they managed to smooth out the jitters, but the output was often too soft, resulting in mushy, imprecise articulation. Both approaches, ultimately, required a painful amount of post-processing time and money before they were fit for a commercial release.

Jang pinpointed the root of these failures: flawed source data and the nature of AI models themselves. When a character pronounces bilabial sounds (like 'm', 'b', or 'p'), the necessary lip-closing action isn't properly captured in standard training datasets like SMPL. Furthermore, because the same sound can be spoken with varying mouth shapes, AI models tend to 'average' the results, churning out a muddy, mumbling mess of an animation.

To break this cycle, the team at NC AI developed 'VARCO Face Sync,' leveraging diffusion models and transformer architecture to generate animation directly from audio. It’s a handy tool: you can take side-quest dialogue or NPC lines, convert them to TTS, and have the animation generated automatically—all exportable as Unreal Engine assets. To tackle the 'dirty' data problem, they even built their own facial capture rig to record specific bilabial movements, ensuring the AI actually had something accurate to learn from.

VARCO Face Sync also employs a retrieval-based audio conversion technique, which maps various speakers onto a single reference dataset. This effectively filters out noise, replacing any input audio with a clean, recognisable voice. To avoid the uncanny valley of a character having, say, a male upper lip and a female lower lip, they used identity embedding to keep speaker characteristics consistent. They also focused on collecting data only from segments where emotions were clearly defined, acknowledging that it’s simply too much to expect a human actor to maintain high-intensity performances for hours on end.

The end result is a system that reduces variance and boosts stability to the point where it can be dropped straight into commercial cutscenes, bypassing the need for manual QA or artist intervention. When a writer plugs a script into the system, the animation is born instantly, drastically shortening the pipeline. As Jang put it, the goal isn't just quality—it's stability: 'Ensuring consistent stability is paramount, and above all, we need to avoid burning extra time and money on manual post-processing.'

Jang isn't resting on his laurels, though. He outlined four areas for future improvement. First is using video generation AI to build training data. Why rely on limited motion capture when you can generate hundreds of hours of data and cherry-pick the best bits? Second, he pointed to AI-driven automated rigging. Manually setting up rigs for complex facial expressions is a bottleneck that keeps character expressiveness within a very rigid box; AI needs to handle this heavy lifting.

Third, we need better emotional nuance. Simple labels like 'happy' or 'sad' just won't cut it for film-quality performances. By using LLMs to label video data with complex directorial notes, we can coax more diverse, natural expressions out of the models. Finally, he envisions the integration of face and body animation. Once your lip-sync is perfect, a stiff, lifeless body becomes glaringly obvious. The ultimate goal, he says, is a digital twin or digital human technology where facial expressions and body language are generated in perfect, harmonious unison.