Creating Believable Talking Animations for Your 3D Characters with AI Voice

This story and the author Tobias Kolodziey are being featured in 80 level. See the interview article: https://80.lv/articles/replica-creating-talking-animations-for-3d-characters-with-ai-voice/

A NEW AVENUE FOR LIP-SYNC ANIMATION: INTRODUCTION TO FAST TRACKING

AI VOICE LIBRARY INTO FACIAL ANIMATION

I’m a Character Artist and Content Developer for Reallusion’s Character Creator for over 5 years now and I was always looking for an easy way to breathe life into my creations. As I had worked in the CGI and animation movie industry for a couple of years before, I know how time-consuming and cost-intensive character voice casting can be. You would need to hire a voice actor, do the sound recording, edit the voice, import into your software, then (frame by frame) edit your character’s talking and facial performance. Voice acting and editing is one job, animation is another job. That could mean weeks of work, and thousands of dollars for the project—which is not something that every creator can afford.

I was lucky enough to test out the new collaborative solution provided by Reallusion and Replica. Replica is a company that specializes in providing AI voice libraries. Thanks to Replica, I now have a huge library of natural-sounding voices—always at my fingertips. Then I found out that the AI Voice Actors Plugin for iClone is opening a whole new avenue for animators: You simply just type the text for your avatar, one-click send to iClone, then your avatar will be animated with automated lip-syncing.

Having given it a deeper review, I decided to write this article to show other creators how this solution is a real game-changer. I hope all of you will find this useful and perhaps change your viewpoints of how to create lip-sync animation forever.

Tobias Kolodziey (TokoMotion)

My name is Tobias Kolodziey and I’ve been working in the animation & CGI industry for several years now. As a character artist, I started Studio ToKoMotion in 2011 to develop and produce my own ideas. Since then I also had the opportunity to work with clients on some small but interesting projects, especially in the field of edutainment. You can see more of my works in Artstation https://www.artstation.com/tokomotion

In this behind-the-scenes article, I will give you a quick overview of how the new Replica plug-in for iClone will help you to bring your characters to life in just a few clicks. Please make sure you have the (free) Replica plug-In installed onto your iClone software. For more information about the plug-In installation, and other helpful hints and tricks, please visit the official Reallusion product page at: https://www.reallusion.com/iclone/ai-voice-replica

You will also need to have the Replica Studio software installed on your computer in order to connect iClone via Replica Link. Please visit https://replicastudios.com to create a (free) account and download the software.

Check out the following tutorial video, and learn the workflow:

LET’S START ICLONE

But now let’s start and see what Replica for iClone has to offer. To open the plug-in in iClone, go to MainMenu > Plugins > Replica Studios > Replica Link. Now you can select the character in your scene via the Replica Link and it is ready to receive the voice from Replica Studio.

IN REPLICA STUDIO

So just switch over to Replica Studio Sandbox and type in some lines of dialogue, then click Select a voice… to choose a voice from the Replica voice library.

You can scroll through the library and preview the voices or use the provided filters to search for specific features of the voice. A total of more than 45 voices can be found here.

After you’ve selected the proper voice you just need to press Play to let Replica Studio generate the voice for your character’s dialogue. This provides a quick way for you to review different voices.

Try to play with the “Style“, “Speech Rate“ or “Pitch“ to further adjust the voice to your liking and save the take to export it to iClone.

AUTO-GENERATE FACIAL ANIMATION STYLES

After you’ve clicked Export to iClone in Replica Studio, iClone will ask you to select an Animation Style and Strength Level that will be applied with the generated voice to your character. There are a bunch of animation styles for different moods from ”Anger“ to ”Surprise“ available for your character’s facial animation. Click Apply and let iClone do the rest.

When finished, your character will have a cool-sounding voice, great lip-sync, and a base facial animation. Here you can already see your character talk with a basic facial animation matching your selected talking style.

If you are not satisfied with the animation style or strength level you can always change it using the Expression Loop presets that can be found under Content > Animation Template > Face > Expression_Loop. Needless to say, you can create your own facial expression clips, but this library gives you another level of automated preset motions.

FINE-TUNING WITH ICLONE’S BUILT-IN ANIMATION TOOLS

Please also make sure to check out iClone’s built-in animation tools, Face Puppet, and Face Key to finetune your character’s facial animation afterward. It’s easy to add some more facial expressions for the whole face or just certain areas like eyes, brows, or mouth/lips with these tools.

Face Key is the perfect tool to create individual mouth/lip shapes for certain letters/words of your character’s dialogue to make it look unique.

Use the Muscle, Expression, and Modify sliders to create special morphs and to emphasize certain letters or words. Additionally, you can manually adjust the auto-generated visemes of the voice track to acquire the perfect lip sync.

Finally, create somebody animations for your character, add props and effects to your scene to bring your character to life—it has never been quicker and easier!