Natural MetaHuman Control using iClone

Intro

My name is Petar and today I’ll be showing you my workflow of animating MetaHuman with the latest iClone tools. As iClone has updated to version 8.1, there are plenty of new animation features to improve your MetaHuman Animation. In this showcase, I am building everything from scratch, but you will see my sharing with the use of Rokoko SmartSuite and learning how to deal with motion capture data. Let’s see what we made and work toward that.

Scene preparation in Unreal Engine

Before we jump into motion capture we should do some block out of our scene or have already the final scene where we’re going to place our animation. We made a simple scene with a few things in the background, and our main props which include a chair, a table, a few lights, and a laptop. It’s good to have some plans for what you are going to capture and how the environment is going to interact with it.

Set-up: Use the Metahuman Kit

Prior to jumping into iClone, be sure to download and install the necessary plugins. Reallusion provides you with the LiveLink plugin and Dummy models which are retargeted MetaHuman bodies. Just copy and paste dummies in your iClone content folders and copy and paste Livelink folders into your Plugin folders inside the Unreal project. The next thing on the list is to set up our MetaHuman blueprint so it can receive real-time animation data. There is a step-by-step tutorial on the official Reallusion youtube channel so be sure to check that out. Once we are done, we are ready to animate.

Motion recording in Rokoko

Once done with a motion capture session, we made a few tries so we can pick the best one for the job. We will use Rokoko to post-process effects on our data to get a much cleaner export to iClone. I always use Locomotion and Drift fixes just to be sure that weird noise in data is processed. Once we are done with the effect we can export our animation as FBX. iClone also provides a Rokoko Profile for iClone which allows us to stream data directly onto our character.

Overview: use iClone as your main animation software

Before we do any work, let’s see the iClone user interface and tools we’ll use. On the left, there’s Smart Gallery and Content Manager. It is an intuitive and streamlined library for managing the various iClone 3D files such as models, animations, and everything associated with a project. On the right side. The Modify Panel contains adjustable parameters that belong to the selected object. It contains the main and powerful animation tools that we’ll use. Timeline is your main playing area, where you can mix animations, set keyframes, filter, sample, and so on. iClone 8 got major timelines upgrades and my favorite is the ability to have independent channels for every bone and morph target. This comes in handy when you want to polish your animations. The great thing that works with timelines is Curve Editor. For the ones who are not familiar with curves, it’s basically just a different interface of keyframes and a much more initiative way to inspect and edit your animations once you grasp the concept.

iClone 8 Highlights: Animation clean-up

Let’s talk about Reach Target, one of my favorite tools. Instead of setting several motion layer keys to reach a character’s head, hands, feet, or target object, you may use a Reach Target key to easily accomplish this animation. What it does is, when you set a target, iClone uses IK from your body to align animation.

In the timeline, you can set the transition type and duration when the reach target will be activated or released. In our project, we use it when our character pulls a chair, lays down their hands on the table, or grabs the laptop.

Next, we use the Curve Editor to inspect some jitters in movements and try to locate them. Using curves you can easily spot areas that need fixing. We delete the selected keyframes, set tangents, and smooth out a little bit. Be careful not to smooth too much, you may get a really generic movement then. To edit animation you can head to the modify panel and use the Edit Motion layer. A window will pop up with a bone picker and FK and IK options. Click on bone, select mode, and adjust animation. To spot pop-ups in the Curve Editor, it takes a little bit of practice but the basic principle is searching for spikes. Curves should be mostly smooth with some little noise. Any sharp spikes that break patterns are errors we need to fix.

In the end, we can animate fingers. iClone provides you with a great number of poses called Gestures. You double-click on them in Content Manager and they will apply to your body. You can use more Gestures, mix them, and set transitions for smoother animation. Once you are done animating fingers, you can adjust them manually to fit around your props.

Facial animation: animate lip-sync and facial expressions

There are a number of different tools to animate our faces. I’d like to start with mouth first, so I’m gonna use the one and only, AccuLips for lipsync animation. AccuLips feature for you to convert voice to readable text, and align the text to the audio waves to ensure the correctness for generating accurate visemes. Once I imported my audio and AccuLips will generate the text from me. I’ll modify and fix some words. After the talking animation is applied, you can still edit it and tweak individual visemes and their strength, smoothness, and moods all while being in the timeline.

Once I’m finished with AccuLips, I laid down a layer of live facial animation. For that I’ll head to the motion live plugin, connect my iPhone and capture the animation. Be sure to make a mask to exclude the mouth and jaw since they are influenced by the AccuLips. I’ll also set some smoothness and record my animation. This will give me a great start for animating the rest of the face.

For additional editing, you can use the Face Puppet System or Face Keys Systems. Face Keys are basically manual keyframing with an awesome bigger interface. They can be used to emphasize some crucial facial expressions while using a puppet system is great for a neck, in this case. Using a puppet this time, you play the animation in real-time and move your mouse cursor to increase or decrease the strength of the movement.

Bonus: AccuRIG

New stuff worth mentioning is Reallusion is the new AccuRIG: it lets you easily rig any character. Why is that cool? Because Reallusion expects iClone for animation and AccuRIG has a library of high-quality animation called ActorCore, allowing you to purchase motions, put on your character and mix and edit animations in iClone. iClone is really becoming a one-stop solution for animating characters!

iClone 8.1 Highlights: Transfer animation data to Unreal Engine

Once we finished the animation, it’s time to get it to Unreal Engine. There are a few options you can use, like exporting FBX or using the iClone Unreal LiveLink plugin, the one we are gonna use it now. Open take recorder in Unreal, select your actor, and record. It’s a really fast workflow. In Reallusion’s iClone 8.1 update, there is a major update. In the past, you could encounter some frame drop issues while using Live Link. Quite frankly, juggling between apps not only increases CPU loading but also causes a drop in productivity. However, with iClone Unreal Live Link starting to support timecode sync, it makes sure the motion data is always sent out in the full frame. Making the recording process less painful.

Making cloth simulation

When the export is finished, our animation will be saved. For this showcase, we are gonna use Simulation on our clothes so we need to export our MetaHuman so we can import a Marvelous Designer. We’ll make a new level export whole blueprint of MetaHuman. Then we open Blender where we’ll import our MetaHuman and all other animated props that we’ll be needed for collision.

Lastly, I export everything from Blender as one alembic one and import it to the Marvelous Designer, run the simulation, and export as OGAWA Alembic.

Conclusion

Reallusion’s iClone is truly a one-stop solution for Unreal MetaHuman animation. From my experience, it is best that you have a clear concept of how you’d like to tell your story and start with a nice script. Once you have passed the concept stage, then you will need to learn how to build your technical pipeline with the needed skills. Luckily Reallusion had incorporated all its official tutorial and learning materials online at a combined portal called Reallusion LEARN. To help, let me also share the webpages that I have mentioned in this page, so everyone can also learn and create as I did!

Animate MetaHuman | https://www.reallusion.com/iclone/live-link/unreal-engine/metahuman/default.html
Rokoko SmartSuite | https://mocap.reallusion.com/iclone-motion-live-mocap/rokoko.html
Animate Facial Expression (Face Key and face Puppet) | https://www.reallusion.com/iclone/3d-facial-expression.html
iClone | https://www.reallusion.com/iclone/default.html
Curve Editor | https://www.reallusion.com/iclone/animation-curve-editor/
Try iClone MetaHuman Kit | https://www.reallusion.com/iclone/live-link/unreal-engine/download.html