Free Blender AutoSetup Plugin Instantly Sets Up Characters from Character Creator to Blender
Martin Klekner has always dreamed of creating animations for believable CG characters from captured actor performances. He got a taste of such technologies while working on cinematics for Kingdom Come Deliverance by Warhorse Studios. However, he was still left wondering whether talking animations could be done by one person at home using affordable software.
Martin Klekner is a freelance director and CG artist from Prague, Czech Republic. He specializes in Blender and is an instructor on CGBoost.com
Complementary workflow between Blender, iClone, and Character Creator
Next-Gen Character Creation and Animation
Martin was very interested in iClone’s capabilities, including animating faces with LIVE FACE tool, generating lip-sync with the new AccuLips, and sending everything to Blender with just a few clicks.
The first step was creating the human base meshes in Character Creator. The new CC3+ character base capabilities allow for more realistic eyes and have a new realtime hair collection. Also, using the options provided by the SkinGen texturing tool was a must; layering great-looking skin details onto the base texture. The simplicity and automation of the process were the key for Martin to make everything in a matter of mere hours.
“Reallusion tools are allowing me to not only create great-looking human meshes, but also to immensely speed up the entire workflow. With the new Skingen and Hair capabilities, CC3+ models now meet (and exceed) all my quality requirements!”
– Martin Klekner
Easily generating believable, fully-rigged charactersOne-click transfer from Character Creator to iClone animation
Once the character’s look was final, the next step was the seamless export to iClone. The fully-rigged character was now ready to be animated using both mocap data and facial animation from the LIVE FACE app. The new step Martin tried was using the AccuLips option, generating automated lip movement based on a single voice clip. It provided a base for the rest of the facial animation.
“Lipsync, something I’ve always dreaded, has become a simple and enjoyable task for me with AccuLips and LIVE FACE.”
– Martin Klekner
Martin recorded his own facial performance using the iPhone LIVE FACE app. He layered animation tracks for eyes, brows, mouth, and head movements, blending everything together and letting iClone automatically smooth the tracks with the new Smooth option. This removed the jerky facial movements one usually gets. Finally, the extensive ActorCore library was a great source for choosing a body animation for the final animated result.
Precise auto lip-sync animation directly from voice input.Best of breed multi-pass iPhone facial motion capture
AutoSetup to Blender
With the animation done, it was time to export everything to Blender. The format he used was the FBX with Blender preset set. Importing to the 3D tool was as easy as using the freely available CC3 add-on. With a single click, the FBX was automatically set up in Blender, ready to use both in Cycles and the realtime renderer Eevee, with fully editable materials.
Adjustable digital human shader for Blender Eevee render
Eevee was used to render out the final shot, which meant realtime preview and thus, speeding up the workflow. Thanks to the automated tools by Reallusion, Martin was able to make this shot in a single day.
Create true-to-life facial animations with a game-changing workflow that includes accurate lip-syncing, emotive expressions, muscle-based key editing, and a best-of-breed iPhone facial capture. Furthermore, a new set of scan-based ExPlus blend-shapes for iClone 7.9, gives unmatched, comprehensive facial performances.
Powerful Tools Come to the Aid
Whether humanizing realistic digital doubles or dramatizing stylized characters, iClone provides time-saving tools that can help anyone direct their 3D actors’ facial performances to maximize productivity. These tools and features come in the form of AccuLips, Face Puppet, ExpressionPlus (in combination with expression preset), Face Key Editor, and ARKit.
Automatic lip-syncing, which is provided by AccuLips, gives the most intuitive and effective way to generate procedural, editable talking animations, by allowing all types of users to easily create accurate, natural, and smooth-talking performances. Face Puppet lets you easily start with 23 predefined facial animation profiles that cover diverse personalities and emotions, with the addition of 7 ExpressionPlus profiles for subtle performances over eyes and mouth movements. All of these profiles allow an artist to animate facial parts like never before! In addition, one can effectively tone down the animation strength for realistic humans, or increase the strength for exaggerated stylized character expressions. It’s also possible to humanize characters with advanced facial behaviors like rolling the eyeballs while the eyes are closed, sucking in the lips, biting the lips, chewing with jaw movements, opening the jaw with lips sealed, poking the tongue in the mouth, licking the lips, and even simulating silent talking.
Once facial expressions are created through lip-syncing, facial mo-capping or puppeteering, one can fine-tune facial features by using the Face Key Editor. These facial keys will be overlaid and blended with existing expressions and stored in the Timeline track for further editing. Three types of expression adjustment are provided: Muscle panel, Expression presets, and morphs sliders. Within the Muscle panel, one or several effective regions can be selected and the user can mouse drag to drive the corresponding muscle movement along eyelids, eyes, brows, cheeks, mouth, lips, and nasal area. Muscle panel includes options for symmetry, head rotations, and adjustable sensitivity to accommodate subtle to drastic movements. Muscle panel also supports both Standard Mode and ExPlus mode (new in v7.9). Expression presets allow for quick jumping between different emotional presets, or access to ExPlus for 4 additional ARKit expression sets, including tongue movements, along with adjustable strength levels to match production requirements. 60 Standard and 63 (52+11)ExPlus blend-shapes can be directly accessed via morph sliders, which not only gives complete control over expression details but also allows for input of extrapolated strength values (+/- 200 ). This makes it easy to observe what blend-shapes are being triggered with expression presets and through the Muscle panel. Furthermore, blend-shape names and values can be inspected when exporting to external 3D tools.
iClone provides the most intuitive way to generate procedural, editable lip-syncing animations, allowing all types of effortlessly created, accurate, natural, and smooth-talking performances. The highly expressive, scan-based ExPlus blend-shapes are now one-to-one mapped to iPhone ARKit signals, making the data ready for post-layer editing via the iClone Face Puppet and Face Key Editor.
Lip-syncing that Works
AccuLips is a brand new lip-sync technology that allows characters to achieve smoother and accurate lip-sync results. This technology can save a massive amount of time when animating dialogue scenes, because AccuLips detects and extracts text and viseme from audio, or can be made to import prepared scripts for precise lip-syncing. Natural speak is achieved with the co-articulation design, and with further fine-tuning of every word’s viseme and strength level. AccuLips contains an English dictionary with 200,000 default words, that are easily customized by adding new words, or making changes to the existing entries.
The new iCloneAccuLips system can automatically generate accurate text data and correct viseme timing from an imported voice. This new design gives users a highly manageable approach to professional lip-sync animation. This new method lets you generate text from audio, while automatically aligning words to their corresponding voice sections. If the system is further aided with a text script, then one can get a more accurate text-to-voice alignment.
With enhanced mouth shape for each viseme, the lip-sync result is more precise and natural for both realistic and stylized characters because AccuLips is designed to simulate real human speech behavior. For true-to-life talking animations, each mouth shape takes on the qualities of other mouth shapes that precede or follow them. i.e., articulators either anticipate the next sound or carryover qualities from the prior sound. “Schwa(/ə/)” is the weak vowel sound in some syllables that is not emphasized in words. AccuLips also include slightly opened mouth shapes when there’s a break between words and sentences and mouth movements are governed by pairing 8 lip/dental shapes, 7 tongue positions, and mouth widths; all blended with varied time curves to represent more natural transitions between a total of 240 possible phoneme combinations. When we pronounce words, the preceding phoneme will be affected by the following phoneme, therefore the viseme of the preceding phoneme may be different, like “key”, “cool” and “car”. The same treatment is applied to succeeding phonemes where they are also affected by the preceding phonemes, like “chips”, “kick” and “lick”.
Although it’s good to see all the detected visemes correctly align to each word, highly frequent lips and jaw movements can cause unwanted mechanical behavior. The smart down-sampling tool can effectively alleviate the judders by reducing less prominent lip keys, just like how humans tend to slur in a effortless manner. Phoneme pair design is the foundation of iClone’s lip-sync system, and the 7.9 update takes out abrupt transitions between certain viseme pairings, and optimizes transition curves for smoother lips blends, especially when talking fast. The strength of lips and jaw movement varies due to when and how people speak. It can be influenced by speed or situation like whispering is different from yelling, and mumbling is different from enunciating. The Talking Style presets give you the quick setup over lips, jaw, and viseme strengths optimized for different talking occasions. iClone lip-sync technology realizes the character lip-sync animation with an easy and intuitive process. The system will align the visemes of phonemes to what the character says, and allow users to further fine-tune.
There are several flexible ways to import voice for characters, and they include the following:
Record the voice directly in iClone, so that the voice clip will immediately show.
Use text-to-speech to have the voice generated according to what is typed.
Import an audio file, so that the system can generate their corresponding visemes in the timeline.
When importing voice files, the system will automatically generate a voice track with corresponding timeline visemes, and with AccuLips there will also be a text track. Users can intuitively adjust the voice and animation, and the selected viseme with the Lips Editor. Use Lip Options to adjust the Smooth Options and Clip Strength of a selected clip. One can also separately fine-tune smooth level for lips, tongue, and jaw.
iPhone Face Mo-cap
iPhone’s facial motion capture works withiClone Motion LIVE by using the exclusive TrueDepth camera system powered by sophisticated high-res sensors. The iPhone tracks face with a depth map and analyze subtle muscle movements for live character animation. The Reallusion LIVE FACE app enables the iPhone to live-stream captured facial mo-cap data directly to a PC or Mac, transforming the iPhone into a powerful 3D bio-metric mo-cap camera.
The TrueDepth Camera projects and analyzes more than 30,000 invisible dots to create a precise depth map of the face, and analyzes more than 50 different muscle movements to mirror expressions. Furthermore, it is highly sensitive to details, like rotating eyeballs and tongues; works with different skin colors and extreme light conditions. Runtime calibration offers on-the-fly calibration without pausing the face tracking process, with support for WiFi, USB, and Ethernet connection. Users are recommended to use an Ethernet connection in order to eliminate all potential lag for the optimal facial mo-cap performance.
Designed by animators, for animators – Reallusion’s latest facial mo-cap tools go further, addressing core issues in the facial mo-cap pipeline to produce better animation. Starting from all new ARKit expressions, to easily adjusting raw mo-cap data as well as re-targeting, to multi-pass facial recording and essential mo-cap cleanup – we’re covering all the bases to provide users with the most powerful, flexible, and user-friendly facial mo-cap approach yet. The biggest issue with all live mo-cap is that it’s noisy, twitchy, and difficult to work with, so smoothing the live data in real-time is a big necessity; because if the mocap data isn’t right, or the model’s expressions aren’t what you want, and the current mo-cap recording isn’t quite accurate then you need to have access to the back-end tools to make adjustments. Finally, while you can smooth the data in real-time from the start, you can also clean up existing recorded clips using proven smoothing methods which can be applied at the push of a button.
Markus Sosnowski – 2D Animator / Content Developer
Markus Sosnowski
Finding popularity for his first original nursery rhyme written for his young daughter, Markus Sosnowski made the decision to create a music video for YouTube with the help of Cartoon Animator, which he now has a loyal following for his collection of children’s music videos.
In his interview with Reallusion, Markus tells how Cartoon Animator, along with his passion for music and illustration, helped him realize his dream of becoming a successful story teller for children.
“I would never have dreamed that I would be able to produce such good quality videos without having any drawing skills myself. As a beginner with Cartoon Animator, you can start animating right away.. I can make cartoon videos in this quality without ever having studied animation or art and without a large team behind me.”
Markus Sosnowski – 2D Animator / Content Developer
Matt Hickinbottom – Owner of Musical Music Pictures
Matt Hickinbottom
Matt started in film and television at the age of 12 as a child actor on children’s television. Being a musician, he soon landed a freelance job writing regular music and jingles for children’s television and Disney across Europe. Wanting to develop his skills further, he learned how to shoot and edit and has become a sought-after producer, director & director of photography. Up until recently, he shot regular promos for SONY pictures with some of his work winning Promax awards.
In his film work he has worked all over the world from Canada and the US, across Europe and as far east as Kazakhstan. He has also created numerous cinema adverts for companies such as Merlin Entertainments, across their LEGO Discovery Centres, Sea Life Centres and Warwick Castle attractions.
‘Emily, the little match girl’ started over 25 years ago when Matt started to write a stage musical based on the Hans Christian Andersen fairy tale, ‘The little match girl’. This current production is aiming for a Christmas 2021 release and stars the voices of Bafta, Laurence Olivier and Annie Award winner ‘David Bradley’ (best known for playing Argus Filtch in all the Harry Potter films & Walder Frey in Game of thrones) as well as Olivier Award nominated ‘Lesley Joseph’ and many more.
Matt started working with Reallusion’s iClone in 2020, as the pandemic made it one of his main tools for creating animated content, where he employed everything from iClone animations to Character Creator, and Perception Neuron motion capture.
“Reallusion’s tools have been the key to unlocking my imagination, allowing me to show everyone else what’s been inside my head for so long. With the lockdowns, it’s also been the one set of tools that’s allowed me to keep working and creating without limitations.”
Matt Hickinbottom – Owner of Musical Music Pictures
Greetings, my name is Peter Alexander, a 3D Generalist, and freelancer who works heavily with iClone and Character Creator.
iClone’s Facial Animation Technology includes facial expressions, lip-sync animation, and facial mocap which provides real-time facial animation with facial capture equipment. AccuLips provides a very intuitive way to generate procedural, editable lip-syncing animations, allowing users to easily create accurate, natural, and smooth-talking performances. It detects and extracts text and viseme from audio, or imports a prepared script for precise lip-syncing which saves a massive amount of time when animating dialogue.
In this article, I’m going to run you through the process of using AccuLips to create lip-sync animation for my Laughing Buddha. I’m also going to show you how to export your animation from iClone to Blender for Eevee rendering.
To keep the focus on AccuLips, I’ve already done the character’s movements using elements from a previous project. You can see the lip-sync animation tutorial in the following video:
Prepare Recorded Voice MP3 File Compatible with iClone
For the character’s audio track, I recorded my voice using Audacity, an audio manipulation application and altered the audio’s pitch and reverberation. After editing the track according to my needs, I save it as an MP3 file, which is compatible with AccuLips and iClone.
Import and Edit the Audio in iClone
In iClone, go to Animation and Create Script, choose AccuLips, and import the saved audio. Select Generate Text. AccuLips will analyze the audio and attempt to approximate the speech into text. The words that can’t be identified will be in red, much like a spell check. You can manually correct this for a better generation of visemes.
Adjusting Text for Better Visemes and Modifying Visemes
When you apply the AccuLips data to the animation, it will produce the various visemes to accompany the audio. I then go through the timeline, and if I want a sound to be emphasized or replaced with the lip-sync, I click on the viseme indicator and adjust it. The key point is that you want to emphasize visemes at certain points during the animation. If the dialog emphasizes a sound, the animation should as well. You can keep the lip editor open as you go through the timeline, to make things easier to adjust.
Export the Audio and Character from iClone
If you choose to render in Blender, you’ll want to do two things. First, export the audio from the range of the video, then select the character and export the FBX from the same range. This will ensure that the audio and animation are in sync.
Importing Character into Blender via Auto Setup
In Blender use the Blender Auto Setup, located in the CC3 tab, to import the FBX file as a character. In the CC3 tab, click the Range from Character option to set the proper animation range for the timeline.
Importing the Audio into Blender
To import the audio, go to the Video Sequencer, press Shift+A, then select Sound from the menu. Or you can go to the Add menu at the top of the Video Sequencer section and add Sound from there. This will allow you to select the previously exported audio file. You can select the Playback panel on the timeline to enable scrubbing so you can hear your audio as you go through your animation timeline. If you don’t hear the audio, open preferences, go to System, then check your Sound settings.
Saving the Animation as a Video File
To save the animation as a video, go to the Scene tab, go to Output, and choose to save it in a video format rather than as an image. Also, remember to choose an audio encoding option. If you plan to do some compositing, you can set up a green screen or save this as a video with transparency (RGBA).
Rendering to Video File in Blender
Now with everything set up, choose Viewport Render Animation, rather than the full render (depending on your rendering preferences). This will render things as they appear in the Eevee Preview window.
Examples of other lip-syncs and facial animations:
Greetings, this is Peter Alexander. I’m a digital freelancer specialized in illustration, graphic design, and 3D modeling.
I’d like to share how I take advantage of Blender’s powerful modeling features in creating clothing and accessories, then send those assets to Character Creator to build up a reusable asset library. Character Creator supports skin weights transfer function and other optimizing tools to make the process easy.
In this article, I am putting existing assets through the pipeline for refinement, so you will have better ideas about the Character Creator-to-Blender pipeline. I will use a free ‘Blender Auto Setup’ add-on to simply the import and export process.
If you are interested in how I built up clothing and accessories from scratch in Blender, please check out this full-length video.
Depending on your needs, export either a clothed character or an unclothed character in the pose you plan to work within Blender. If you export with a bind pose, make sure your character is in the same pose when you re-import your clothing and accessories.
As I was going to be re-importing all of the items I had exported in an optimized state, I deleted his existing clothing after I exported the clothed character.
Import Character in Blender via Auto Setup
In Blender, using the CC3 tab provided by the Blender Auto Setup add-on, import your file using the ‘Import for Accessory’ option. This is similar to the other options, but is optimized for asset editing and export.
Tweak Clothing and Accessories in Blender
Although not necessary, I fixed a few visual settings, such as the opacity blending for the smoke I had previously created. Doing this is good practice, especially if you plan to render within Blender at some point.
Make any adjustments, edits, or even create new assets from within Blender, depending on your requirements. In my case, I mostly adjusted a few clipping issues and added some edge loops to assist with clothing deformations.
Export Clothing and Accessories via Auto Setup
After making the adjustments or creating the digital assets, export the item back through the Blender Auto Setup add-on as an accessory. If you’ve imported the character as an OBJ, you’ll be provided with an OBJ option to export back. I believe if you’ve imported an FBX, that option will be provided.
Bring in Assets with Skin Weight presets
Import the items one-by-one through the Create > Accessory option. This will import the item in approximately the same position relative to the posed character as it was created in Blender. Sometimes you will have to do some placement adjustments.
By default, the item is an accessory (unskinned geometry), and you can part the item to various bones for accessory (such as in the case of this character’s cigarette and smoke, which I parented to the Jaw bone). Clothing should be skinned through the ‘Transfer Skin Weights’ option.
Optimizing the Assets with Hide Mesh Tool and Skin Weight Editing Tool
Clothing that conceals significant geometry should be optimized to auto-hide that geometry when fitted to the character. This can be done in the ‘Hide Body Mesh Tool’. This not only optimizes the visuals during clothing deformation, but it will also allow you the option of deleting hidden mesh automatically during exports into other applications.
Finally, try various poses to see where issues with skinning weights should be adjusted. In my case the clavicle was pulling the collar of the coat too harshly, so I assigned some of the vertex weights to one of the upper spine groups.
Send Character Back to Blender for Eevee Render via Auto Setup
If you plan to render out the character using Blender’s Eevee engine, you can export the character with animation or in a pose using the FBX export options. I tried the option to delete the hidden mesh during export. This eliminates the ability to re-import back into Character Creator, lessens the need to adjust the mesh after it’s been imported.
You can import the FBX file to Blender through the ‘Import Character’ option of Blender Auto Setup add-on, which will automatically translate Reallusion’s Digital Human Shader and parameters to Blender Eevee and Cycle. You can further adjust SSS skin, eyes, teeth, and hair dynamically.
Comprehensive 3D Hair and Beard Component-based Design for Any Hairstyle Imaginable.
The embeddedSmart Hair system is designed to provide the most realistic, yet performance-friendly hair, brows, and beard for digital humans in games and films. The card-based hair meshes are intricate in detail, economical in texture resolution, and efficient for real-time rendering. The highly flexible color gradients and specular variations are powered by the Smart Hair shader. Design components allow you to freely assemble different custom hair and beard elements into unique hairstyles, while the beard and brows can dynamically conform to any facial movement.
A Plethora of Choices
Smart Hair’s component-based design offers an easy and effective way to manage 3D hair, brow and beard assets. Users can add select hair parts and accessories, combine elements into groupings, and compose them together to form a cohesive look. Group and Style items can be replaced with a single click to quickly test out different variations. Imported assets assigned to a corresponding facial area automatically conforms to the underlying surfaces . The beard is composed of mustache, goatee, sideburns, soul patch and other accessories. On the other hand, hair is divided into base, top, rear, bangs and accessory parts for optimal mixing. Use the component design system to alter character hairstyles with different fringes, bodies, and ponytails. You can also fill in tiny hair and wispy fuzz around the hairline and back of the neck.
Creating a multitude of natural hair looks is possible with four powerful controls: Ombré (root and end color), Balayage (highlight color), Base Layer (vertex color), and Reflection (flow map). With these potent effects you can easily create chic or stylized hair dye, create silky smooth hair reflections, simulate silver gray, or parched look and feel. Blend with vertex color (predefined in hair meshes) for enhanced depth effect. For natural layer contrast, you can use lighter vertex color on upper layer meshes, and darker vertex color for layers underneath. Create subtle gradual hair color fades using Root and End colors, usually from darker tones at the roots to a lighter tone towards the hair tips for a classic ombré style. The strand color pattern is defined by the grey-scale Root map, and supports different blend modes and strength controls. The Highlight Color effect is the highlighting done with hair dyes that we see in reality; Two highlight colors can be blended to create more natural looks, with theID texture map controlling the effect. The natural reflections gives hair a healthy lifelike look and feel . One can adjust the Specular strength, alter the Reflection Flow, or disrupt the reflective pattern to emphasize unique hairstyles.
The Smart Hair system is designed for performance: the card based meshes are small in size, fast to load, and efficient for real-time animation. One complete hair set is merely hundreds or thousands of polygons, while using optimal texture sizes : hair at 2K and beard at 1K. By utilizing minimal resources you can achieve high visual realism with sophisticated coloring, and adjustable reflections using the real-time shader.
The Smart Hair Beard and Brows can perfectly conform to animated facial surfaces during real-time performance. It works in real-time for facial morphs, lip-sync animations, facial motion capture, and intensive expression change with the Face Puppettool. One can also fine-tune the shape and position of the facial hairs with the Mesh Editor. To top it all off, facial hairs are exportable, at any stage, as FBX blend-shapes for game engines and 3D rendering.
Reallusion’s Smart Hair system is compatible with major 3D hair creators like Ornatrix (for Maya, C4d, Ma, ...), XGen, and Blender hair plugins – as long as they can export card based hair meshes in OBJ or FBX format, and generate the texture maps needed for the shader effects. You can convert existing hair assets as Smart Hair components, or upgrade the rendering with the Smart Hair shader by adding required effect maps. Smart Hair supports direct shader compatibility with Unreal Engine; with one-click you can transfer hair assets through the Auto Setup plugin, while keeping the identical feature adjustments consistent between the two programs. For Unity and other 3D tools, you can choose to bake shader effects to PBR textures for rendering.
Smart Hair’s beard and brow elements are designed to move along with their corresponding facial vertices – however when bringing in new items which have vertices detached from the facial surface, users can use the Vertex Assignment tool to define the target following area to avoid mesh stretches, or mesh penetration, when animating the face.
For more information, visit the Smart Hairlanding page.
Neil Biondich is a writer/director/animator, based in Los Angeles. He grew up in Milwaukee, Wisconsin where he studied art, film, music, martial arts, and roleplaying.
Neil began animating in the nineties, co-founding the company, Sik Puppy Studios. The company’s tech progressed from Amigas running Lightwave, to SGI’s running Alias Power Animator, to finally, PC’s running Maya. Neil had left animation behind for more than a decade but was drawn back in when COVID broke out and he began to study Virtual Production methodologies. The real-time raytracing capabilities of Unreal Engine blew him away and reignited his passion for animated storytelling.
Neil founded Kuden Creative with a charter to develop original, high-concept content for film and television. His sci-fi feature screenplay, KITTY, was a finalist at the 2020 Austin Film Festival and is now in pre-production with Neil set to direct.
His Pitch & Produce project: Chiburi employed real-time tools from Reallusion including Character Creator, iClone, LIVE FACE mocap for iPhone, with Rokoko Smartsuit Pro, and Unreal Engine.
Chiburiis a story about breaking the mold that society tries to keep us in. The plight of a girl in medieval Japan trying to rise above her station is perfectly synonymous with today’s struggles for equality. The character Sister just wants to earn her adult name. Unfortunately, her father will only award a name to one who lands a blow on him with a sword. Females are not allowed to touch swords. So, while Sister is raised watching her father train her brother to become samurai, she sneaks off into the woods to secretly practice what she observes.
“Reallusion was the exact toolbox I needed for my project. Each application tackled a different aspect of my pipeline and it all worked together flawlessly. They manage to bring high-end character modelling and animation into the reach of artists and filmmakers (non-3D gurus) without sacrificing quality.”
Mixing 2D with live-action has been a popular format for movies and TVs over the years. The obvious benefits of this style include that it produces visually realistic results and makes the film more engaging. However, creating such a piece can sometimes be painstaking for video animation makers. Today we have invited Chris Meyer-White to show you how to easily accomplish video compositing of 2D animation with live footage using Cartoon Animator 4.
A Brief History of 2D and Live Action Animation
My name is Chris Meyer-White. I have worked in Home Entertainment for 15+ years, in both the US and the UK, and on projects for Warner Brothers, Disney, BBC, Paramount, and many other studios. The most time of which being at Abbey Home Media which was the UK’s leading independent home entertainment company for lots of well-known children’s shows, Care Bears, Topsy and Tim, Peter Rabbit, Sooty, Horrid Henry, Fireman Sam and more.
Before I start showing you the workflow of my creation, let me give you a quick introduction to 2D and live-action video.
There are many different styles of animation, 2D, 3D, stop motion, and more.
One of my personal favourite styles has to be when 2D and live-action are mixed together. This is sometimes known as hybrid animation, and it’s a technique that has been around for over 100 years.
The earliest example of this style I could find was used in the 1918 series, Out of the Inkwell, followed soon after by the Alice Comedies series by Disney. The Alice Comedies were a series of live-action/animated shorts that followed Alice (a live-action girl) having adventures in an animated world with animated 2D characters.
This was a technique that Disney seemed to love as they created many classic movies mixing live-action and animation in films such as Bedknobs and Broomsticks, Song of the South, Mary Poppins, and more.
Dot and the Kangaroo is a classic story from Australia that sees the main character Dot, lost in the Australian Bush. Dot and all the other characters she meets in the film are 2D and the backgrounds are either live-action film or photos and, as a 5-year-old, this style really captivated my attention and creativity as it was so different from anything I had seen before. The quality is now a little dated, but it still has a magical charm and such a mesmerising style for a young child to see.
One of the most famous films to mix 2D and live-action has to be the classic 1988 film, Who Framed Roger Rabbit, directed by Robert Zemeckis, starring Bob Hoskins and Christopher Lloyd as live-action characters, along with the other stars of the film, Roger Rabbit and Jessica Rabbit, and many more famous cartoons as 2D characters, in the groundbreaking film where cartoon characters and people coexist in the same world. And, for me as an 8-year-old, this film was absolutely mind-blowing and really cemented my fascination with both film and animation.
I have a lot to thank the makers of Dot and the Kangaroo and Who Framed Roger Rabbit, as it was these 2 films that made me want to work in animation and film.
With Cartoon Animator, you can create similar styles, easily and fast.
Over the years, I have used many different software packages to animate – After Effects (using DuiK and Joysticks and Sliders), Adobe Animate, Moho and more. Cartoon Animator (CTA) is the only software package that I’ve used that was easy to use straight out of the box. For example, the templates that come with CTA are a fantastic starting point for anyone looking at character animation, the free motions have you animating in no time.
Another reason why CTA is fantastic is that there are always brilliant new additions. The 360 head turn feature is a game-changer, and the 3D-2D animation motion feature is ground-breaking.
Now with CTA 4.5 you can export your animated 2D characters as transparent .mov files and use them in external effects and editing software such as Adobe After Effects, Final Cut Pro, DaVinci Resolve, Motion, HitFilm, and more.
CTA supports both transparent video (MOV) and image sequence (PNG) export. If you choose to export your characters as transparent png sequences, which is great, however, it would result in thousands of files (as each frame was a .png file). Rendering out as a single transparent .mov file means you can keep the files down, and help keep your project tidy, which really helps the animation pipeline.
Exporting as a transparent .mov file really is as simple as clicking a button. When you go to render your video, make sure you select Apple ProRes 4444 and this will keep the transparency. Then import into your software of choice and submerge your character into your scene – it’s that simple.
After Effects
For me, as I always use After Effects, you can simply export to After Effects with the plugin, and select MOV Video, and, like when you render as ProRes 4444, your project will import into After Effects and, rather than a PNG sequence, it will import as one .MOV file with transparency.
Final Cut Pro
As a video editor, I love when you can make a smooth pipeline between software and design, and the new CTA feature of exporting transparent .mov files is a welcome addition to my animation and editing pipeline. Like you can do with After Effects, you can easily import your animation sequence with transparency and insert this into your video project. You can easily have your animated characters interacting with video footage in no time. Having this feature I’m sure will help all video editors.
When using this technique in FCPX, my pipeline is as follows:
Design in Photoshop, bring my character into Cartoon Animator as a PSD, animate in CTA, then simply export as a QuickTime using the ProRes 444 codex, import my character’s QuickTime file like I would with any other video clip, and layer up the layers to help embed my character into the sequence.
HitFilm Express
I have to be 100% honest. I’ve been a die-hard Adobe user for years, and am a big fanboy of the creative cloud. I’ve been using After Effects for 15 years now, and will never really use anything else… However, HitFilm Express is really something people should seriously look at when it comes to compiling together sequences from footage created in CTA. I haven’t really played around with HitFilm much, but I have seen many CTA users use this and they have created mind-blowing sequences. So I would highly recommend looking into this as a tool to use with CTA. Oh, did I mention, IT’S FREE!!! Well, the basic version is, however, you can upgrade and I would highly recommend doing this.
With HitFilm, as with After Effects and FCPX, I simply import my transparent QuickTime file and layer up the scene with additional elements. However, you can do lots of special effects (like you can with After Effects) such as shadows, lighting, and more. Definitely check it out as it’s a fantastic tool to help add an extra level of sparkle to your animations.
Conclusion
What I love about Reallusion, the company making Cartoon Animator, is that they are always updating the software, constantly making it better for users to create fantastic animation sequences.
Exporting as a transparent QuickTime for Mac and PC is a great addition, especially now you’re also able to export to After Effects as a QuickTime as well as a .png sequence via the AE plugin. This, for me, helps improve my pipeline, and I’m sure, helps other users quickly and simply create broadcast-quality animations. Download the free trial and see for yourself.
