The AI MoCap Plugin released by Reallusion for iClone is so simple to use that it almost needs no beginner’s guide, but, as is the case with tech, it can be confusing to get started. We will take a look at the simple process of translating videos into a complete motion capture motion that can be used on any iClone or Character Creator character.

While the plugin appears to be very forgiving of the quality of the video, it never hurts to develop and use best practices, so we employ that habit from the beginning. This doesn’t mean that you shouldn’t try to capture a motion under less than perfect circumstances, it just means to be mindful of what the mocap translation has to work with.

I can remember using video motion capture a long time ago when apps like IPI Soft were released. While it was a bit revolutionary, it did have its drawbacks. A lot of jitters for one thing, and jitter was hard to eliminate either live or in post. Another problem was that it literally translated everything from posture to performance with zero forgiveness.  This required preplanning and realizing that you would have to deal with cleanup no matter what.

There was a lot of “noise” back then, and mocap suffered for it. Not so today with AI assisting in the process, so it’s basically just editing.

Stacking Motions

I’m not sure if this is the right term, but what I mean is, do not just limit yourself to one short motion before converting. Stack several motions together with a second or two in between so you can edit the motions by breaking them on the timeline and saving them as separate motions. In my testing, I recorded far more video than needed or allowed, which is 60 seconds of video.

Since it is a video, we can edit that video for total control over what is processed. This makes planning paramount. So, try to get as many motions as possible from each clip conversion.

Here is an excerpt from Google AI Overview for more information:

For the iClone Video Mocap plugin…  …the actual motion capture generation is limited to 60 seconds per task, with longer source videos automatically trimmed or requiring you to trim them first to get the best results. Each 60-second segment processed costs points, so maximizing the 60-second window is efficient. 

The AI Squared tutorial later in this article shows an example of stacking clips to get the most out of each video for mocap conversion.

A Quick Look at Features

From the Reallusion website:

• Capture Challenging Pose Transitions: Track body turns and posture shifts with unwavering stability.
• Detect Upper Body and Finger Movement: Produce talking performances with precise hand gestures
• Reproduce subtle and delicate performances: Capture nuanced movements and aesthetic details in every action.
• Precise Footstep Detection: Maintain accuracy throughout complex and fast-paced foot movements.
• Multi-Angle Tracking: Accurate side, back, and rotational tracking.
• Handles Minor Camera Shifts: Counteracts camera shakes and unsteady shots.
• Works Well with Real World Videos: Handles dim lighting, busy backdrops, and patterned wear.
• Multi-actor Interaction: Handle group footage with a single-performer focus with isolated framing and tracking focus.
• Turn talking videos into animations: Transform ordinary monologue videos into dynamic full-body animations, or focus on upper-body performance for precise dialogue delivery.

As stated in the last bullet, the plugin works well with real-world videos and lighting, but it never hurts to limit distractions when possible and develop some good practices when you have the time and space to do so.

Lighting and Background

If cameras are involved, then lighting is important. As I said earlier, this is not meant to discourage a quick capture in less-than-ideal lighting, but we do need to provide stable, bright lighting without washing out the video performance if possible. It doesn’t have to be perfect. I used home lighting for my capture tests in this article and had no problems with it.

Again, not an ironclad requirement, but simplifying your background for the captures can’t hurt. It’s less noise for the plugin to work with. I had a set of French doors with solid brown curtains, so I set up in front of them. It was flanked by two table lamps that helped with the lighting and was already empty of objects, being a doorway.

Acting (Practice)

You can always wing it, but like most things in life, it’s better to prepare for it. Decide what you are going to do. Even simple motions need some practice as you might want to even consider overacting some mundane tasks to get enough motion for “acting”. Everyday motion is not necessarily the same as an “acting” everyday motion for dramatic effect. I used primitive props to keep my hands anchored to a certain area and avoid drift.

Recording

Footage, lots of it. This doesn’t mean taking hours of footage, but multiple video takes will result in a bigger pool of video footage to draw motions from, so don’t be stingy with the takes. Get several takes of each motion and pull out the best footage to edit together for the final mocap clip in your favorite video editor. Like most animations, how much effort you put in now will show in the final result. You want the best of the best for your mocap.

Cameras Used

The cameras I tested were the Samsung A54 Phone and an old 1080p Logitech C922 Pro webcam on a phone tripod, and both worked flawlessly. Since one of my work computers is by that doorway, I used the USB webcam so the video would be recorded directly to the desktop. Neither camera is premium, with the phone camera being much newer and more advanced, but a bit more effort to offload.

Bottom line is I could use the cameras I already had… down to the oldest webcam with no problems. This also gave me the bonus of working with equipment I already knew and trusted. The webcam was also an advantage in that its files were much smaller than the Samsung phone recordings, like 50 megs vs 200 megs or more.

Example One: Operating a Console

For this one, I pulled out my music keyboard and stand so I would have a reference for the console, as I didn’t want my hands all over the place. What I had in mind was a good filler motion for wide use while being simple enough for a test. Having a generic console motion was one I could certainly add to my motion library.

I was thinking of sci-fi scenes where busy motions in the background are needed, like a command center or a ship’s bridge. Having a motion that randomly moves a character’s upper body and limbs in relation to a control surface can be used in a lot of situations. My biggest problem was not treating it like a musical keyboard. I didn’t want to play music… I wanted to operate a transporter or security console, so I had to fight the urge to “keyboard” it as an instrument, but it was a solid reference point.

Example Two: Piloting a Plane

This example was possibly the first motion I mocapped with IPI Soft way back when. It was memorable enough that I used the same technique for reference props.  Instead of a Christmas Tree stand I used in this capture I used a bathroom plunger in the original version, but the result was a reference point to anchor my arm and hand motions to. The plunger was a bit short, and having just cleaned up from Christmas, the stand of our fake tree worked just fine, so my arm didn’t “drift” all over the cockpit or possibly outside of the cockpit, requiring cleanup.

So, there I was, in the living room on a short stool with one hand around one half of a Christmas tree stand and the other hand ready to flip invisible switches, twist some knobs, or whatever I needed to look like I was doing something. The shot involved quick camera shots from outside the cockpit, so all I needed was motion, and it didn’t have to be complex motion.

Example Three: AI Squared

In this test, I used Midjourney-generated videos from images as the video for the AI mocap conversion video. It’s a very simple process of describing what you want in a prompt to Midjourney, like:

full body shot of a young lady talking to another person off-screen on a white background.

A very basic prompt for a basic motion with good lighting, a clean background, and a black screen pause between the AI-generated clips. Like the other videos, I assembled and edited the clips in Vegas Studio, so only this video needed no trimming as it was less than 60 seconds.

I stacked a long dance motion with a few individual motions, like talking on a phone or speaking to a person off camera, with black screens between them to easily identify the break between motions. This actually had the effect of making one of the motions start off-center as the previous motion ended on the other side of the screen and then started from the far side, creating a jump.

The capture needed some cleanup, but it was minimal positioning that didn’t amount to much.

The Process in Review

This is where I thought I would be putting together a major tutorial, but it turns out that only a few basic steps are needed.

1. Drag in or load the video clip.
2. Trim the clip to the desired length.
3. Generate the motion.
4. Apply the motion.

It is just that simple to use.

Behind the Scenes – Quickmagic Integration

All of the magic happens behind the scenes, but here is more information from Reallusion.

• Workflow: Users can import any video clip, and the Video Mocap Plugin uses QuickMagic’s cloud-based engine to generate editable 3D animation.
• Capabilities: The system supports full-body tracking, upper-body gestures, and detailed finger tracking.
• Pricing: The service typically operates on a pay-per-use basis via “DA Points” for video processing. $2.50 US per motion as of this writing.
• Cleanup Tools: Since AI mocap often has artifacts like foot sliding, the integration allows users to use iClone’s native tools (Foot Contact, Curve Editor) to refine the data instantly.

Summary

This was another one of those assignments that I had no idea how it would go, and it turned out to be a complete no-brainer with practically no learning curve.  You follow the four steps mentioned in the conversion process, and the AI takes care of the rest. I have to hand it to QuickMagic and Reallusion for a seamless integration of tools that not only makes sense but is very convenient.

If AI in general weren’t so new to many of us, there wouldn’t even be a need for a “getting started” tutorial, and I don’t think in all these decades I have ever said that about animation software. The environment and lighting are very forgiving, making spontaneous filming of certain motions worry-free, but again, it’s always good to start with “best practices,” so we create a positive habit from the start. Control the shooting environment as much as is practical to maintain continuity of work.

All in all, this was a very positive experience and another boon for beginners to be able to create custom motions with something as simple as a phone camera or webcam, then turn that into a motion!

With AI knocking down hurdles and empowering almost anyone to be a filmmaker or storyteller, the time has never been better to get in on the newest evolution of digital animation and art.

MD McCallum – WarLord

Digital Artist MD “Mike” McCallum, aka WarLord, is a longtime iClone user. Having authored free tutorials for iClone in its early years and selected to write the iClone Beginners Guide from Packt Publishing in 2011, he was fortunate enough to meet and exchange tricks and tips with users from all over the world and loves to share this information with other users. He has authored hundreds of articles on iClone and digital art in general while reviewing some of the most popular software and hardware in the world. He has been published in many of the leading 3D online and print magazines while staying true to his biggest passion, 3D animation. For more information click here.

WarLord’s Workshop