My name is Alessandro Cucinotta. I am from Italy but based in Switzerland now. My career began in Turin at a service company at IBM Group. Since then, I’ve always been passionate about motorsport and aviation. Eventually, I started developing graphics and software for flight simulation as a hobby.
In 2004 I left IBM to embark on a professional adventure in the flight simulation sector, developing airports and airplanes in 3D. I collaborated with Microsoft on developing the 3D Virtual Cockpit of the military Jet F18.
To easily populate our worlds we have been using Reallusion’s ActorCore characters and motions, along with iClone to make our airports more immersive with fantastic results.
Q: Hello Alessandro, and welcome to the Reallusion Magazine! Kindly tell us about your story at Simultech.
In 2016, I moved to Lugano, Switzerland. There I founded my first company, Simultech. At that time, I released my first video and simulation of the Memphis International Airport (KMEM). Not only did I simulate the airport apron, but I also imported several ground control avatars inside my other renders to give the audience a realistic walkthrough experience at the airport tower. Our company specializes in add-on production for the renowned Microsoft Flight Simulator. Simultech cooperates with FSDreamTeam since its foundation, and its popular packs include airport packs such as JFK or LAX or the last Kord V2, or even GSX level 2.
Today, we develop airports and software products for Flight Simulator 2020 with a few collaborators. We are among the first ones to include animating human characters in the world of flight simulation, making the ground environments more alive and immersive.
Q: How long did it take for you to familiarize yourself with iClone, Character Creator, and ActorCore? Any suggestions for beginners?
I started using iClone and Character Creator about one year ago. I was looking for a product that could help me create characters optimized for a real-time in-flight simulation world. After some quick comparison, I discovered that Character Creator was the best tool for me. The possibility to vary morphology and characters’ accessories with tools for the reduction of polygons makes this software a must for time optimization.
In my workflow, iClone represents the main actor. With it, we create 3D animation cycles utilized in real-time in the simulator. The ease of creating 3D animations and the possibility of exporting on my platform, allowed me to realize my ideas with efficiency and in a reasonable amount of time, without relying on expensive motion-capture tools.
Finally, ActorCore just gave developers like me the freedom to preview the final result on the cloud, and I can quickly save tons of time by purchasing the right motion content. Surely I suggest Reallusion’s suite to beginners: they will find everything they need to be immediately productive.
Q: What matters the most when selecting 3D content for simulation games? And what value did you find in Reallusion content and mocap animation?
In the flight simulation industry, it is important to maintain high frame rates, and the ActorCore actors are created with this in mind. The animated human characters are the right balance between performance and graphics fidelity, perfect for real-time applications. For my company, ActorCore represents an impressive time-saving solution for creating characters.
“Luckily with the launch of Reallusion’s ActorCore line, we now have a library of actors developed specifically dedicated for real-time simulations that are incredibly realistic with optimized textures.”
Alessandro Cucinotta – CEO at Simultech Sagl
Q: We love the way you utilize ActorCore and Character Creator in your Flight Simulation add-ons. Would you tell us a bit about what will be expected in 2022?
The success of the Microsoft Flight Simulator has attracted many new content companies, raising the level of content. Our studio put a lot into the graphic representation of characters to make more credible and real the interaction between the pilot and his surroundings. With our latest Replacement Assistance Trucks add-on, we used a lot of characters intended to carry out normal ground operations, which took place at the departure and arrival of the aircraft.
For the next couple of months, we are planning to develop boarding and deboarding of passengers. We plan to use ActorCore heavily, so we can simulate a deboarding of 500 passengers in real-time from a 747!
Greetings, my name is Peter Alexander. I’m a 3D generalist who specializes in utilizing Character Creator with Blender character creation pipelines. I’ve had the privilege of working on character projects to demo the new features coming to Character Creator 4 (CC4).
As someone who has worked with 3DXchange for both Character Creator 3 and ActorCore avatar optimization, I was immediately blown away by how much of an improvement the CC4 facial profile pipeline is. Previously, the 3DXchange pipeline was geared more toward marketplace optimization. Character Creator 4 has features that speed up the process at least ten-fold, and makes it more accessible for the average user.
Unique Personality v.s Generic Expressions
Having been developing more than 100 character products for the past few years, I always look for interesting ways to express my characters’ stories. The general facial expressions can be easily done by any artist, however, to make your own character stand out, it’s essential to give them unique personalities, especially when you plan to animate them after the 3D modeling.
After trying out CC4’s new features, I selected two characters from my previous work: a Stylized Vampire and a Hunchback. My initial impulse was to optimize the expression presets for the rigged character, which has been a standard optimization requirement. As I dug into the tools, I realized that I could exaggerate every aspect of a character’s expression, and thus breathe personalities into characters beyond the standard set of expressions.
The vampire character was fun to work on, as he was the most stylized of the original characters in this demo. I drew a little inspiration from Hotel Transylvania for the expressions and personality of the character, and maybe a bit of Fright Night, though the character design was more of an original experiment.
The personality I crafted for the vampire was that of a suave aristocrat, with an over-the-top sinister – but goofy – quality to his character. I focused heavily on his brow, jaw, and teeth – with many of the expressions exaggerating his fanged smile. Of all the characters, this one probably has the most personality, as the style I chose welcomed cartoonish exaggeration.
CC4 simplifies the way of shaping personalities
In my past experience dealing with large-eyed characters, I have often had to rely on using the Correct Eyeblink feature to adjust the eyelids for blink animations. If this isn’t enough to correct the eyes, however, then I can still use Character Creator 4’s expression morph sliders along with the Mesh Edit tool to quickly overcome this obstacle by adjusting the source morph target shapes.
Apart from this, cartoon characters tend to have more exaggerated proportions to make them visually appealing, like this vampire character with a long face and large mouth for example. However, characters such as these usually need to be adjusted to support exaggerated expressions and performances. Due to the slender face of the vampire, the teeth and fangs have a tendency to penetrate the face, and morph sliders would need to be relied upon to reposition and resize the teeth to fix the flaws in animation.
In an effort to enhance the flexibility of the expressions and performances, I’ve prepared quite a few custom morph sliders on the side. Users are welcome to mix and match these CC4 morphs to arrive at unique character expressions which they can truly call their own.
CC pipelines Create Animated Personalities⎯ZBrush, Blender, and more
The character Luffy from One Piece was interesting to experiment with because of his elastic, over-the-top expressions. As the GoZ pipeline became more refined, I was able to exaggerate the teeth and mouth, allowing for most of Luffy’s classic anime expressions.
The CC4 facial expression pipelines offer amazing convenience for manipulating expressions to create unique personalities for any imaginable character. Luffy’s personality was already established, so it was more a matter of utilizing Zbrush to approximate those expressions of his personality.
CC4 custom expression sculpting with GoZ pipeline.
As I continued to use the new tools, I wanted to experiment with a cuter character, so I introduced this cartoon girl to the mix, which I enjoyed working on. I don’t exactly recall the inspiration for this character; I frequently browse stylized designs on Pinterest, and may have picked up some elements from the Boondocks series. I exaggerated her teeth and cheeks to give her a chipmunk-like cuteness. I envisioned her to be cute but rebellious and prone to challenging authority.
I want to emphasize that my facial expression manipulating experience prior to CC4 was limited to adjusting eye blinks and the expression pipeline is extremely easy to use. Those who are familiar with dialing in and baking morphs will find that there’s not much difference, especially if they use GoZ (Zbrush) or the Blender character creation pipeline to make character adjustments. Once you’ve made your adjustments using expression sliders, and the internal mesh editing tools (including the new amazing Delta Mush feature), you can bake your work as a component of the character’s expression profile. In cases involving symmetrical adjustments, you will have the option to ‘mirror’ your work over to the other side of the model.
Having finished the eye animations for one side of the face, I use the Mirror feature to apply the same slider settings to the other eye.
If you’ve worked on models using 3DXchange, you’ll probably understand how time-consuming and tedious it can be to adjust a character profile, especially things like Eye Occlusion and Tearline. The new pipeline doesn’t just save you hours of work, it literally saves you days. Beyond fixing CC3+ characters, you can take a custom-rigged character from Blender or another 3D application and add an entire personality within hours.
This is game-changing stuff. It makes establishing a character’s personality a pleasure and brings this aspect of 3D development to a level much more comfortable and desirable for the average creator. The CC4 tools are not just fun to use, they also empower artists to bring their character design potential to the next level. If you’re already on the next level, CC4 will take you even higher.
Reallusion is hosting its annual 2D competition Animation At Work from now until August 8th, 2022. Participants are encouraged to utilize Cartoon Animator 4’s real-time character-creation and 2D motion-capture technology to create entries in categories including business & commercial, comic & art, and educational. The total prize pool is over USD 15,000 for worldwide 2D artists, with multiple sponsored prizes from XP-Pen, Magix, and Affinity.
Five categories; 12 winning spots, and more!
The Reallusion Animation At Work contest invites participants to animate real-life projects for business promotions, educational/training materials, animated comics, YouTube videos, and more.
Business & Commercial: Use an animated video as a commercial to promote a realistic or fictional idea, product, or service.
Comic & Art: Here we encourage graphic art designers who can turn original static drawings, illustrative art, or comic images into an animation.
Education: Using animated videos for educational purposes such as storytelling, songs for kids, or learning videos.
Vertical Shorts: Create a mobile-friendly vertical 2D short and upload it to Youtube, Instagram, and TikTok. Video length must be 15 seconds above.
VLOG & Vtube: Use Cartoon Animator in your VLOG and Vtuber projects. The animated 2D characters should blend well with the video contents, and help illustrate a story.
Students and Teachers: Reallusion welcomes students and faculty to show your talents and turn your ideas into animation!
The best chance for aspiring artists to enter the contest
Reallusion welcomes worldwide 2D artists to leverage a 2-month free software and free educational resources to participate in this event.
2-month free software: Do not have Cartoon Animator to compete with? Share the contest webpage on your social media, and email us your social share screenshot to event@reallusion.com, you will be entitled to a 2-month free software of Cartoon Animator.
Free webinars: Hosted by Mark Diaz, The founder of 2D Animation 101, TED Talk speaker, and Reallusion’s certified trainer. Mark will guide you on how to get started with Cartoon Animator and master the key skills for the contest. There will be one webinar in June for making business animations, and another one in July for making comic-style animations.
The Reallusion Animation At Work Contest is an event that invites EVERYONE to use their imagination to animate real-life projects like business promotions, educational / training materials, how-to videos, animated comics, YouTube news videos, and more. Would-be participants can freely enter any of the four categories to compete with the 2D animation community.
Cartoon Animator is a 2D animation software designed for both abilities of entry and productivity. You can turn images into animated characters, control characters with your expressions, generate lip-sync animation from audio, accomplish 3D parallax scenes, produce 2D visual effects, access content resources, and wield a comprehensive photoshop pipeline to rapidly customize characters and create content.
The “2020 Animation At Work” contest was a runaway success that consisted of 25 winning submissions, with more creativity and skill in a single contest than we have ever witnessed. Winners not only walked away with big prizes but also got the chance to be featured on the web page and become featured developers on Reallusion content stores.
Reallusion Launches Impactful Character Pipeline – Redefine Design to 3D Animation
Reallusion is again evolving Character Creator and iClone, the most user-friendly and dynamic character animation platform, with a monolithic leap in ability and creativity. The tandem release of iClone 8 (iC8) and Character Creator 4 (CC4) delivers an array of accessible 3d character tools for generating and animating scalable digital humans or humanoids. CC4 and iC8 aim to provide creators with a unified and dramatically simplified solution for character setup and 3D animation.
The evolution supports custom IP and imported models, benefitting users from studios to artists with an illustrious toolset that now can be applied to any character. CC4 and iC8 open the door to more storytellers and interactive designers with a system to provide professional tools blended with ease of access and ultimate power. Take any character or animation to the platform of choice, from AR, VR, or metaverse to games, films, and content for new emerging media.
Apart from a plethora of professional animation editing features, iClone 8 further innovates animation design with a next-level Play2Animate philosophy. One of the elite new tools in iC8 is Motion Director, revolutionizing character animation with a gameplay style of 3D animation and control interface utilizing gamepads, devices, and keyboard controls for characters, cameras, and UI operation allowing ultimate interactivity.
Seamless Integrated Workflow
Massive innovation inside iClone 8 and Character Creator 4 is helmed with the inclusion of new complete integration of 3DXchange that now seamlessly incorporates character setup and animation with drag and drop import, instant or custom retargeting, and a whole new Smart Content management system for fluid workflow.
Smart Content Manager – users can directly download free resources or purchased assets within the content manager. The CC4/iClone8 shared folder design allows the same assets accessible from both programs, keyword searching and custom tagging for quick data access, Content Workgroup for team assets sharing, and the flexibility of central server access or multiple drive storage.
3DXchange seamlessly merges into CC and iClone, importing characters, motion, props, and more with drag and drop FBX or OBJ, characterizing and exporting to all platforms straight from CC4 or iC8.
iClone 8 – Curve Editor, MotionLIVE are now standard features
Total integration of CC4 character updates between CC and iClone
ELEVATE CHARACTER DESIGNS (CC4)
Character Creator 4 goes deeper into character development with bilateral features for a new version of CC4 digital humans and imported humanoids that support all the animation setup tools associated with CC4 native characters. The broader compatibility with any character to become a CC4 character and take advantage of the iC8 animation system is a big change to CC4 that opens the door to 3D artists allowing them to create and set up their characters to come to life through the real-time animation tools of iC8.
CC4 characters have the most customization for animation setup with custom blendshape import and assignment that unlocks the iC8 facial animation and lip-sync for any character model.
Extended Character Realism
The CC4 native character is upgraded with 140 blendshapes from 4D scanning for refined motion capture, face puppeteering, and lip-sync with iC8 AccuLips.
Create a custom face expression blendshape or morph setup from the Expression Profile Editor. Character artists have the freedom to create a unique personality for their own iconic character.
Enliven Any Character Rig
Whether humans, creatures, or props, creativity is no longer limited by the existing CC character base. Any rigged biped models can be imported, characterized, and facial rigged in Character Creator. New features now make any character compatible with thousands of motion assets, ready for natural lip-sync, motion capture, and animation controls in iClone. CC characters can also be optimized for low-poly, high-performance crowd simulation, AR, VR, and Metaverse.
Auto Import From Popular Character Types
Character Creator can auto-detect the imported character type when dragging in an FBX file made by popular vendors. Supported character standards include Maya HumanIK, 3ds Max Biped, Blender Meta-rig, Mixamo characters and motions, and Daz3D G3/G8 characters and motions.
Showcase Full-Motion Characters
The CC Animation Features let character designers see living characters inside Character Creator, helping to check imported character rigs, enhancing cloth skin weight and physics, and testing facial expression and lipsync assignment right after customization.
Character designers can use the turntable to real-time review character art or render it as video. Adjust settings for turn speed, cycles, and turn items. Free 3D turntable bases are included with customizable logo textures.
Glamorous realtime render with a library of realistic lighting, and lens flare effects optimized for the digital human shader.
DRAMATICALLY SIMPLIFY CHARACTER ANIMATION (iC8)
iClone 8 animation delivers a pro editing result with autonomous animation creativity. The Reallusion Play2Animate philosophy and Device LIVE features demystify the traditional animation complexities. Experience the balance of deeply refined editing controls blended with a new intuitive approach to animation. Mixing the best of both worlds between gameplay interactivity and professional animation, access editing tools to layout and lens scenes with devices for camera controls, and animate characters with ultimate interactive and hands-on direction using gamepads, 3d mouse, and assignable hotkeys.
Motion capture and animation editing in iClone 8 are refined with smart controls that assist in footstep/handprint reach-target locking for precise character performances with scenes and surroundings, character-to-object, or character-to-character animation.
Professional Motion Controls
iClone provides powerful Motion Editing Innovations while keeping the process simple and intuitive. Whether users are new to the 3D industry or a veteran of professional tools like Maya or MotionBuilder, iClone’s motion editing features and unparalleled simplicity can out-perform most animators’ expectations.
Automatic root motion alignment between mocap clips
Smooth Bidirectional Blending, and Directional Control over motion clips
Optimal HIK Controls for accurate pose-to-pose editing and IK/FK blending
Multiple Motion Layer Editing, a non-destructive workflow for adding blended motion options
Character-centric Timeline, with switchable track numbers: body part level, body bone level, facial feature level, facial morph level, and IK/FK blend track
Pro-level Curve Editing for professional cartoon performance and blendshape follow-through animation.
Facial Animation Editing
Whether humanizing realistic digital doubles or dramatizing toon/stylized characters; iClone provides time-saving Facial Expression Editing tools that can help anyone direct their 3D actors’ facial performances to the highest productivity.
Works with various facial profiles, Traditional, CC4 Basic 60+, CC4 Extended 140+
Enhanced Face Puppet and Muscle Edit with secondary-level facial feature options
Facial curve editing for facial bones and blendshape animation
Facial-centric timeline-track editing down to Feature Level (63 tracks), and Morph Level (60-150 tracks)
AccuLips lipsync animation works for both standard CC3+ characters and imported Humanoid characters with CC4 facial setup.
Mocap Motion Correction
Motion capture and animation editing in iClone 8 are refined with smart controls that assist in footstep and handprint systems
Reach-target locking for precise character performances with scenes and surroundings, character-to-object, or character-to-character animation.
Motion key sampling, smooth filter, key reduction, and curve editing
Motion Director
The Motion Director system debuts gameplay controls to drive characters and applies motion triggers and dynamic cameras to direct scenes in real-time. Implemented with motion matching technology, the new feature provides smooth speed change, turns, and seamless motion transitions.
Game-like Actor Control – steer characters with a regular game controller, mouse-click waypoints, or conventional WASD keys.
NPC AI Animation – give autonomous idle, walk animation on the path, in the zone, or follow an animated object
Camera Switch – be able to switch up to nine camera views when steering the actor
Macro Recorder and Sequencer – record device input events and replay in user-defined sequence, or bake to timeline
Behavior Expansion – allow adding new behavior to actors
Custom Hotkeys & Device LIVE
Intuitively engage in directing and editing with Live Switching, Trigger Effects, Slider fader, Jog Dial timeline scrubbing, Drone Camera navigation, and so much more. Almost all functions in iClone can map to custom hotkeys for assigning devices that interact as physical interfaces for ultimate scene control. Users from Maya, Max, or Blender can also choose their preferred mouse-keyboard convention for viewport control.
Camera Switch & Navigation
Motion & Timeline Editing
3D Object Transformation
Lighting & Perform Triggers
Directing features above can be triggered by regular XBOX or PS game controllers, 3DConnexion SpaceMouse, MIDI slider, Monogram Creative Console, Tourbox, Stream Deck, and other Windows compatible devices.
Other New Features
There are hundreds of requested features being answered in CC4 and iClone8, here are some other noticeable highlights.
Flexible frame rate systems, allow frame-accurate timeline editing and video composting
Enhanced performance and new productivity design, new scene manager with modular Collection design, be able to toggle collection visibility, make group transformation, material and mesh instancing, and project merge.
Now there is more power than ever to the Reallusion pipeline from both CC4 and iC8 including features enabling creators to generate and export characters and animation in FBX or USD for the most popular platforms. Export CC4 and iC8 scene assets and characters for path-traced rendering in Omniverse, game output and virtual production in Unreal and Unity, roundtrip editing with 3D digital creation tools, and making next-level metaverse avatars.
iClone and CC seamlessly work together as one big design-to-animation character platform and greatly ease the in-and-out productivity with many other 3D applications.
Instantly inject believable souls to your digital actors, a total of 25 minutes long, 146 facial sequences meticulously produced at different conscious levels and emotional states.
Quickly enliven a crowd simulation scene with 14 automatic walking styles, and 11 idle motion styles. Supports both player control mode and ai-control mode with Motion Director.
Andrew Macdonald – Executive Creative Producer at Cream Productions
Andrew Macdonald
A serial innovator throughout his filmmaking career, Andrew has always been the first to bring a new camera or technology to set. This led him to co-found Cream Productions’ digital department with Cream CEO David Brady in 2015. This venture has grown from one person incubator to an award-winning full XR production department.
Andrew has become well versed in concept definition, interactive UX design, directing motion capture, game development, and innovation of XR production workflows. He is currently directing an applied research machine learning project on digital humans at Sheridan’s Screen Industries Research and Technology (SIRT) centre.
Cream Productions is a Toronto-based Producer of documentary television, specials, and digital media that has created over 20 pieces of original XR content including 360 3D series, AR apps, and interactive volumetric VR, for networks, studios, agencies, and brands all over the world. They also developed SurvivormanVR a single-player interactive VR game for Quest and PCVR that challenges players to survive dynamic, sophisticated, real-world life-or-death experiences in remote, wild, natural locations, under the watchful guidance of real-world expert and TV celebrity Les Stroud.
For SurvivormanVR, the team at Cream Productions contacted SIRT Centre for the best pipeline to create digital doubles which led them to the Dynamic Digital Human system (DDH) that is employing Reallusion’s Character Creator (CC) software.
“Reallusion software is affordable, gives you multiple output options, and because its procedural it allows creators to iterate quickly.”
Andrew Macdonald – Executive Creative Producer at Cream Productions
iClone, Omniverse Audio2Face Deliver Accurate Singing Animations for Rap Music Video
Film director Solomon W. Jagwe, the co-founder of Sowl Studios and the creator and director of The Adventures of Nkoza and Nankya, shows how he achieved accurate lip-sync animations for a rap song using Character Creator, iClone, and Omniverse Audio2Face.
The Challenge
As a 3D animation storyteller and filmmaker, one of the most challenging aspects of the process of bringing out a convincing performance from a performance actor, has to do with making sure the lip sync is well done and that the soundtrack matches the lip movement so the viewer is not distracted and instead focuses on the story. Now, if you are an indie filmmaker on a budget and with a small animation team, that challenge is multiplied many times over, because you have to work with limited resources, time and funding. That means that you have to constantly be on the lookout for ways to efficiently accomplish your facial animation tasks.
When NVIDIA Omniverse and iClone partnered to create a solution that made the facial animation workflow way more efficient for indie filmmakers, I knew I had found a solution to bringing to life the story I had in mind.
The Final Result
You can check out the music video showing the successful result. Below are the workflow I took to create the Maze BeatZ Rap music video, using Audio2Face for the facial animation, using the Perception Neuron Suit to capture the body mocap, refining the facial and body animation in iClone, and rendering the final animation in Omniverse with Path Tracing.
How I Created the Song
The first step I took to create the Maze BeatZ video, was to come up with the rap song. I am not a rapper, but thankfully, there is already a rap track that comes with Audio2Face. The one I used as a starting point is labeled rap_NV_Project1_JDB.wav, which can be found under the Audio Player tab wave source. The initial problem I faced was that I needed my rap star to be a female artist, and yet the rap track was that of a male voice. A quick solution was taking a trip to Adobe Audition, and applying a pitch shift to the rap track, and voila! I had my female rap artist voice.
Another aspect I needed to add was a rap beat to the empty rap song. So I partnered with a young and very talented Ugandan artist, whose awesome music I chanced up during my music research on YouTube. His name is Andereya Baguma. Once the song creation was done, my attention turned to bringing my female rap artist to life.
How I Created the Female Rap Artist
For the starting point of the body of the character, I used Alisha, because I needed a middle aged Afrocentric character. I also made sure to use the Export version of the character because I needed to send the completed model to Omniverse for rendering.
I wanted my rap star to have a cool rap outfit, so I opted for the Leather Outfit. Again using the Export version, which is highly recommended if you ever need to export your character for use outside of Character Creator or iClone.
To round out the cool outfit for my rap star, I picked up some realistic looking glasses and a beautiful hat . I stayed in the realistic realm because I needed the character to look great when rendered in Omniverse.
I used Character Creator to assemble the model and prepare it for the trip to iClone so I could use Motion Live and the Perception Neuron Studio suit to capture the body mocap, and also prepare the rap artist for the facial animation creation using Audio2Face.
I absolutely love working in Character Creator because of the multiple ways you can fine tune your character to create a unique result. The smart gallery plugin in Character Creator makes it very easy to import all your purchased items, and apply them to your model inside of Character Creator. I was able to further refine my model using the morph and sculpting tools.
After I completed refining my rap artist in Character Creator, I exported the character as an iAvatar to iClone. You can send the model directly from Character Creator to Audio2Face, as a USD file, but I needed to make a trip first to iClone since that is where I was going to do all my animation both for the body and face.
How I Animated the Face and Body of the Rap Artist
To get my model ready for 3D animation, I loaded the iAvatar character I exported from Character Creator, into iClone and set up some initial lighting and also imported a stage from Daz Studio, where the rap artist was going to perform.
The cool thing about Character Creator characters is that they come fully ready for both facial and body animation inside of iClone. Reallusion offers several truly powerful plugins in the Motion Live platform, that extend the animation power of iClone. I relied heavily on the Perception Neuron plugin for the body mocap.
I used the Perception Studio Suit from Noitom, to capture the dancing sequence of the rap artist. My 15 year old daughter volunteered to play the part. After recording her rap performance, I saved the iClone mocap files from the Perception Neuron Suit session, into the Motion folder so I could repurpose them and apply them to my iAvatar after creating the facial mocap in Audio2Face.
Once I had the base body mocap done, I then exported my Character from iClone as a USD file, to Audio2Face. I made sure to check the “Include Omniverse Audio2Face Mesh” check box in the Export USD dialogue box. That step is extremely crucial because it offers Audio2Face the ability to recognize your character’s face so you can use the AI to create the facial mocap from an audio file. I also made sure to select the render mode. I chose to export both the RTX and the Path Tracing render mode as 2 separate versions of my character, so I could have the option to preview the animation in real-time as well as render the character using the Path tracer in Omniverse.
I imported my rap artist character into Audio2Face and began the process of using the built in AI to transfer the facial motion capture data derived from the rap vocal wave I mentioned in the earlier steps of this workflow.
Audio2Face Highlighted Steps
I was blown away at how well Audio2Face was able to use the vocal of my female rap artist to create the lip sync animation! Audio2Face also offers several powerful tools under the Emotion tab, that make it possible to add nuanced lip movements after the ai applies the data from the audio file onto the characters head.
After applying the Facial mocap to the head of my character in Audio2Face, I exported the data as a json file, with just the lower part of the face.
I imported the JSON file with the lower facial animation from Audio2Face, into iClone, and applied it to the Rap Artist character using the Omniverse plugin.
Audio2Face does an awesome job of capturing the nuances of the voice in the audio track, but one thing that it wasn’t able to capture, at least as of the writing of this article, is the tongue movement and the eye expressions that one needs to create a convincing facial performance. I believe a solution is in the works for future versions of Audio2Face, but for my video, I turned to the amazing animation tools inside of iClone to add those much-needed facial animation nuances.
One of those awesome tools is called AccuLips. I was able to take advantage of the text-to-speech power of AccuLips to create an added layer of facial animation which included tongue movement.
I was able to further refine the lip sync animation with the Lip options, which gave me access to the individual visemes to fix any area of the mouth movement that needed improvement.
The final step of the facial animation was to add some blinking and more custom upper-face, facial animation to give my rap artist some personality. I used an iPhone X to capture myself performing along to the rap vocal. For capturing the data from the iPhone X, I used the Live Face plugin in the Motion LIVE inside of iClone together with the Live Face iOS app from Reallusion.
I was also able to use the iClone mocap editing animation tools to adjust the Perception Neuron data. I loaded the mocap file I had saved from the session with my daughter, and applied it to my character. iClone made it so much easier to edit the mocap from the Perception Neuron Studio, captured using Axis Studio from Noitom.
With the Edit Motion Layer tool, I was able to fix the fingers of the character and add the grasping of the microphone that she used in the final video.
How I Setup and Rendered the Final Video in Omniverse
Once I was happy with both the facial animation and the full body animation, I selected my rap artist character and exported her as a USD file from iClone to Omniverse. I exported 2 versions, one for RTX and another for Path Tracing. Doing it that way offered me the flexibility of choosing whether to render using Path tracing or Real-time rendering in Omniverse.
Exporting as Path Tracing has the advantage of providing Omniverse with a character that has a beautiful skin shader that supports subsurface scattering (SSS). The Character Creator models come prepared with a beautiful skin shader that looks great when rendered with path tracing in omniverse, but they also take longer to render compared to the RTX version.
I loaded my rap artist into a scene I had prepared using a set I exported from Daz Studio as an FBX, and loaded into iClone. I used iClone to set up the audition stage, and also placed the character in the focus area, which I then used to set up different cameras that I knew I would need to render from once I got to Omniverse. It’s truly awesome that Omniverse supports all the lighting and camera animation from iClone.
I set up some additional lighting inside of Omniverse to bring the stage to life and to add camera movement. To preview the animation before rendering, I set the display in Omniverse to real-time, which made it very easy to iterate and make camera and lighting adjustments.
I rendered the final animation using Path Tracing. Having a good GPU goes a long way toward speeding up your renders in Omniverse. I also used the built-in denoiser to help speed up the renders.
For rendering I used a custom machine from the team at Puget System, that has an NVIDIA RTX 3090. This made a huge difference in turning around the final music video. Rendering at 2K can take up a lot of space, so I recommend having an external SSD drive with plenty of space, especially if you end up with an audio file that is longer than 2 minutes.
Final Thoughts
I highly recommend the Omniverse, Audio2Face, Character Creator, and iClone combo solution as a go-to facial animation workflow in your storytelling journey and I believe they will make a huge difference in your project. So go ahead and give them a try.
NVIDIA offers Omniverse and Audio2Face as free downloads. Realluson also offers the Omniverse iClone Connector as a free download that you can find inside of your Omniverse Launcher. Reallusion offers versions of iClone and Character Creator for you to try out before you decide to purchase. If you don’t have a motion capture suit or don’t own Character Creator, you can use the amazing characters and mocap from ActorCore, the 3D people and mocap animation Library.
The “Winning Tips & Tricks” series covers practical workflows and techniques shared by winners from the “2021 iClone Lip Sync Animation Contest”. To let users see the full spectrum of the iClone facial pipeline, we are introducing projects that are rendered with different 3D engines including; Unreal, Unity, iClone, Blender, and Cinema 4D. Let’s now take a look at Jasper Hesseling’s: “Jessica Rabbit”, combining Character Creator (CC3) / iClone workflow and awesome iClone native render.
Jasper Hesseling
My name is Jasper Hesseling. I’m a solo 3D artist for more than 15 years. Based in Utrecht the Netherlands, I create under the name “Mayonnaise”. Mayonnaise as a name came up because I sometimes have the feeling that I create the visual sauce for concepts and ideas next to the fact that it is a very popular sauce in the Netherlands to emphasize my roots. From traditional 3D model and sculpting to texturing, rigging, and animation, I am an all-around 3D artist that understands how to leverage the advantages of each 3d tool and assemble them into the ideal animated character pipeline. Originally my background is in motion graphics but I transitioned to 3D art and animation because therein lies my passion.
Why I created this showcase
I am a fan of Jessica Rabbit since I was a child. Her charm and radiant character are something really special that I wanted to present to the competition. With the use of my talent and paying details to capture her likeness, I hope my submission can be seen as a tribute to the iconic film “Who Framed Roger Rabbit”. During the competition, I thought my 3D skills would be a good asset and help me bring my A-Game. But I think, in the end, the ultimate goal of this is to test my creativity while having fun!
How I do it with iClone
Step 1: Finding reference
References are very important – I think a lot of people usually forget about them. I found a lot of materials online. Some are actually original still frames from the movie without all the character poses, and I also had very nice finds from some fan art. It was difficult to find some images from the side of the head, so I really need to guess what it’s gonna look like. The program that I use here is called PureRef. It’s pretty handy and you can use this as a window to drop in all your reference materials, or just let it lid somewhere else on your screen.
Step 2: Cloth modeling in Cinema 4D
The dress was poly-modeled in Cinema 4D, and I changed the breasts a bit, pulling them closer together. It’s a basic approach that allows me to get some proper cloth UVs out there. With me following the meshes, the deformation and the weight painting are pretty solid as well. Now there’s an option to actually hide clothing, or hide stuff under the clothing. But I just wanted to be flexible and sometimes it doesn’t really work well when you export because when you hide stuff in the meshes inside of Reallusion (software), then you get a triangulated export so this is not really perfect.
Step 3: Accessories – Gloves, Dress, and the Shoes
I used the hair materials for the glove, and it’s because the hair material has this nice anisotropic and specular going over the surface. This allows you to mimic silky fabric. You can play around with the settings and adjust the shininess or roughness. It also works well with normal maps on the Modify Tab, even on the wrinkling parts.
As for the dress, I used a texture that created myself. I created that in the Substance Designer, and it’s actually available through my website. You can change the color and it works pretty well. I was pleased with the result that actually came out of its shader. It’s nice to play around with the texture, making things shiny and bling-bling.
The shoes are from my own Reallusion webshop, but they came very handy in this case. They are available for you guys if you would like to have them inside your collection.
Step 4: Facial appearance and Make-up
For the rest, I hand-painted the texture out and I am using Substance Painter. But it was pretty simple because it’s so toony, and I didn’t really have to go in there and do a lot of details. Creating a comic style for animated characters is simple, all you need is to match up with the whole reference library that I had on her. There weren’t many details in terms of texturing. But pay attention to some details like the bump map, just to break it up a little bit, and the lipstick is just shiny as it can be.
Step 5: Edit the hair in ZBrush
The hair is a low poly model, and I use sub-division inside of Reallusion so it’s working pretty fine. And you can just use brushes to adjust the hair. Another big tip for you guys is that when using ZBrush, the scaling is pretty large coming from Reallusion, so what you need to do is change the dynamic brush scale to about fifty (the number that I normally I use), and now you get a proper brush size going.
I use the move topology brush, which allows you to move things without moving the rest of them. It just focuses on cross meshes. So if I do like a standard move the whole meshes change.
Rotate the hair, select the Display Properties table and click the “double” button. That will give you a better vision when editing the inner hair. Click the GoZ button and you’ll be back to Character Creator and see the result.
Step 6: Earrings creation and Lighting settings
The earrings were pretty easy, it’s just to wear. What I did is make attributes out of those ones so that they will follow the head. Another thing that is really important is the lighting. Looking at the whole scene, it’s pretty simple. Select the minimal mode so you can see what it consists of. Since it was a lipsyncing contest, I’m just focusing on the lips, and no need to build the whole theater. I turned on the auxiliary light, and we will get some more detail in here. One important tip is to start doing the character poses and block the animation and create.
Step 7: Camera settings and the Environments
Once you’ve completed the lighting, then you can start working on the camera. I started with a medium one. Because the background was pretty minimal, you can see the total view. With a lot of darkness in there, it was quite forgiving to use only basic meshes in the background. I even use the meshes that were inside here to mock up the curtains. I got my inspiration from there and with a little clever lighting, you can actually just use the darkness. Tip: No light is also lighting as well! You can also apply some PBR here. I used the shader, change the tangent type to green color, and I use diffuse color and play around with the settings here. I use an HDRI in the background to shine and get some nice reflections going on in the dress, but the main lights are in the scene as you can see. I use a very strong key rim light to actually create this sole stage sheen a bit like a spotlight that’s following her around.
Final Step: Post-production in After Effects
So once when I was done with the animating in iClone, I rendered the whole thing, and it gave me a PNG sequence. I would have preferred an open EXR 32-bit so you can actually tweak the colors a bit more. But for now, it was okay. And what I did in After Effects is actually bringing it back to 24 frames per second, so that the animation was really looking choppy and fit the whole style a bit more. And I tweak the colors a bit more of course.
Final Thoughts and Conclusions
That’s it, it’s quite amazing that you can actually get results like this in like two weeks of production time. Looking at the whole finished thing in two weeks it’s quite amazing!
Gabriel Escalona – 2D Animator | Lunacreciente YouTube channel
LunaCreciente is a YouTube channel that uses Cartoon Animator to create engaging and catchy music videos, and more, for early childhood learning. With over 8 million subscribers to date, this channel is a shining example of how Cartoon Animator is being used to produce quality 2D animations in record time to draw audiences in.
In his interview with Reallusion, Gabriel Escalona talks about how Cartoon Animator provides him with tools that allow him to work fast without sacrificing quality.
“I’m not a professional animator. And that’s the reason why Cartoon Animator software is perfect for me, because it has premade character motions that I can use and edit. I can simply drag and drop motions on to my characters and the hard part of animating is already done for me!”
Gabriel Escalona – 2D Animator | Lunacreciente YouTube channel
Comparison of Omniverse, Unreal Engine 5, and Maya V-Ray for 3D Real-time Animation
Introduction
José Tijerín is a digital illustrator and 3D sculptor who has created a video game “Dear Althea” to be launched on the Steam. His “We’re Besties” content pack is currently selling in the Reallusion Content Store.
In this tutorial, José shares his way of working with Character Creator and iClone to get fantastic results in Omniverse, Unreal Engine 5, and Maya V-Ray render engines. Then he compares the results and gives his personal opinions on each one of them. This tutorial will help users to discover new possibilities for showcasing their 3D sculpt and animation work.
Prepare A Character
The first step would be to create a character in Character Creator, however, I’m going to skip this step because I have already written an article with a tutorial video on how to create a cartoon-style character.
Now we are going to add movements to this very same character. In order to bring it to life, we can take advantage of ActorCore’s fantastic motion library.
Once we have selected the desired character animation, we’ll just have to apply it to the character and adapt it to the scene we want represented in iClone. In my case, I had to select an animation of a cook stirring contents in a pan and another motion for taste-testing with a spoon. In order to make the 3D animations fit the scene, I had to connect the spoon to the hand, fix the other hand to the pan handle, and adjust all of the movements to fit the character I was working with. Finally, I animated the character’s gestures and eyes to bring her to life, which is without a doubt, the funnest part of the whole process.
NVIDIA Omniversef
Let’s start with the NVIDIA Omniverse platform, an application that is easy to download and completely free to use: Just register an account on their website and download the program. I’m adamant that everyone should try this software at least once due to its simplicity, usefulness, fun, and ease of use.
Once we have it installed, go back to the open project in iClone and look for the USD export button at the top of the program. In the subsequent pop-up window you can configure various aspects of the export, but in our case the default setting will work as they are. Next, click on the Export button to take the whole project along. We can click the Send to Server button to send it directly to Omniverse, but in my case, I’ll just work off of a scenario already prepared in the program.
Export an iClone project to NVIDIA Omniverse (image credit: José Tijerín)
Now, we will export our files to a folder and open the NVIDIA Omniverse Launcher – if not already opened. The Omniverse Create application can be found and launched from the Library menu.
For my project, I just need to load a scene in which the character will be introduced. To do this, go to the File menu and click on the Add Reference option. Don’t worry if the character enters with whitish skin – thanks to iClone’s intelligent export system to Omniverse – all of the skin texture parameters will be adapted to the program.
To apply the textures to a model in Omniverse, we first have to click on the model by clicking either on the 3D model or on its name in the list to the right and apply new material. Afterward, you’ll notice that the lower right window will be filled with the material properties applied to the model. These material properties will be editable after clicking on the material symbol.
Apply textures in Omniverse Create (image credit: José Tijerín)
Applying the textures to the material is very simple, we only have to look for the folder with the textures in the Content window and drag the textures to different sections that make up the material and configure it in an intuitive way.
For other occasions, as in the case of cartoon-style hair, we’ll need a special material. To apply a different material to an object, we have to right-click on it. In the floating menu that appears, click on the Create menu and, inside it, look for the Materials menu. For this example, I’ll choose one of the predefined Presets of the OmniHair material.
Character hair creation (image credit: José Tijerín)
In case of problems with the 3D animation, such as objects in Attach mode not moving correctly, you should go back to iClone and bake the animation. To do this, go to the timeline. In the CC3 Base Plus section, go to the Collect Clip track and select the entire animation track. Once selected, right-click and perform Add MotionPlus tro Library from the pop-up menu. Then find the custom animation in the content menu and apply it to the character. And in the situation where the facial expressions don’t work, you will have to update the programs and re-export the character.
The Three Different Render Modes in Omniverse
Next topic is on the Omniverse render modes, which is really impressive stuff. If we go to the top of the viewport we can see the lightbulb symbol. Click on it to see three types of rendering that Omniverse has to offer:
RTX Real-Time is one we have been seeing so far, which corresponds more or less to the traditional lighting of video games.
RTX Path-Traced is the strong point of the program.
Iray Photoreal is the most accurate of the group, but it is slow and computationally heavy.
One of the main differences between the rendering modes is the rendering time. For my project, the scene is complex in terms of lighting and textures, so to avoid noise in the video, the render times are quite long compared to the incredible speed that Omniverse usually offers.
Omniverse render mode: RTX Real-Time (image credit: José Tijerín)
The RTX Path Traced result is excellent with rendering speed so fast and of such high quality that you can work on the project while rendering without a problem. This gives you the possibility to tweak the scene without having to wait for the render to update; something that can speed up the production by a lot.
Omniverse render mode: RTX Path Traced (image credit: José Tijerín)
Finally, Iray Photoreal is NVIDIA’s bid to become one of the major rendering engines dedicated for the big screen. Compared to other industry-standard render engines, it can be plagued with some small shader problems in certain situations. However, judging by the color transmission present in the left-over food, it tends to give better results than other modes and can be quite impressive for some scenes.
Omniverse render mode: Iray Photoreal (image credit: José Tijerín)
As you may have noticed, unlike the other modes, the rendering of translucent material does not work by default. So to make it work, we have to go to the Iray menu to the Render Settings section to find the Caustic Sampler option. When we activate it, we get a result that is quite impressive, even for a very simple scene.
Comparison of render modes of Omniverse (image credit: José Tijerín)
Render with RTX Path-Traced
However, given the complexity of the operations to be performed to achieve this result, the time required for this mode to completely remove noise and achieve high definition is very time-consuming compared to the other modes. So I recommend choosing the intermediate mode to work in for good, responsive results.Now that we have chosen a render mode, let’s see how to go about rendering a scene. If we go to the Render Settings menu, we can find the Path Tracing rendering menu. Inside we can find two important sections: Path-Tracing and Denoising. To explain these parameters we are going to render a complicated material like the pre-defined material for honey.
Comparison of render modes of Omniverse (image credit: José Tijerín)
In the first section, we find the Total Samples per Pixel parameter. The higher the number, the greater the number of samples and the higher the quality, but at the cost of longer render times.
The second section is the Denoiser, a system that many 3D programs have been using for some years now. This system detects the noisy areas in the render and softens them. However, you have to bear in mind that the more samples you have to work with, the better the results will be.
Now that we have the basics set up, let’s render the animation by going to the Rendering menu and in the drop-down menu open the Movie Capture menu. In this window, we can configure parameters such as frame rate and resolution. Just select the option to Use Current (RTX Path Traced) and set the number of samples per pixel. When we are done we just have to assign a destination folder for the output images and hit the Capture Sequence button.
The final result is quite good and we have obtained it in a very short amount of time compared to traditional renders.
Comparison of render modes of Omniverse (image credit: José Tijerín)
Unreal Engine 5
Downloading and installing Unreal 5 is completely free, although we must have enough space on our hard disk and some previous knowledge of 3D programs to learn the engine.
Once Unreal is downloaded and installed, we are going to import the animations from iClone. To do this it is advisable to put both programs (iClone and Unreal) side-by-side on the same screen to optimize workflow. Once we have opened both programs, we must go to the Unreal Settings menu and click on the Plugins menu. In the pop-up window, type “Live” in the Search section to check if Live Link and iClone Live Link plugins are activated.
You’ll notice that the new Live Link menu resides next to the Details menu. If we click on the Source button in this menu, we can access the drop-down menu and select the iClone Live Link option and then click Ok to accept the port number.
iClone Live Link with Unreal 5 (image credit: José Tijerín)
iClone Unreal Live Link
After everything is ready on the Unreal side, we’ll go back to iClone for a moment to make the connection. To do this, we need to deploy the Unreal Live Link menu by going to the Plugins menu. After clicking on it, one should see the Unreal Live Link menu with the options for Transfer and Link. Make sure the Transfer option is selected and go to the bottom of the window and click on Transfer File.
After waiting a few minutes, we’ll have the imported character with all of its materials already configured. Next, we’ll need to link the animation to the transferred character by going back to the Unreal Live Link window in iClone and clicking on the Link button. Again, we have to go to the bottom of the window and click on the Active Link option. If you come across some problems with the linking process, you can click on the character and go to the Details window to diagnose the errors. Under the Animation section, make sure to set Animation Mode to Animation Plane.
Recording Unreal Animation
Now it’s time to record character animation in Unreal. To do so, go to the Sequence Recorder window where we will introduce the models we want to record into the Actors list. Once this is done, simply press the Record button and, when the countdown is over, press the Play button in iClone. When the animation has finished, we have to press the StopAll button in Unreal.
When we do this, a pop-up window will appear at the bottom right of the screen so that we can open the sequence directly. Now that the iClone portion of the process is complete, we can close the application to focus on the Unreal scene. As you can see, in addition to the models we have imported, the models linked to the animation sequence now appear in the scene. We can remove them from the scene so that they don’t get in the way while we work.
If there are textures that you have forgotten to put into iClone, you can still apply them to the materials in Unreal. If we press Ctrl+Spacebar we can access the project content menu. Just import the textures into a folder and create a material. Double-click on the material to access the Material Graph, a graph view where you can drag the textures and connect them to the appropriate channels.
Record animation in Unreal (image credit: José Tijerín)
Once we have connected the textures, we just have to scale the material and drag it to the Materials section in the corresponding element of the model. Now that we have all the materials ready and the scene prepared, we are going to render the video. In the Sequencer window, we have to click on the film plate button to access the Render Movie Settings menu, where we can configure parameters such as the output format or the size of the frames. When we have finished configuring the parameters we just have to give it a destination folder and click on the Capture Movie button.
The result is really amazing for a video game rendering engine thanks to the revolutionary Lumen technology.
Unreal 5 render result with Lumen (image credit: José Tijerín)
If we remove Lumen, the same scene would look as below because the light would not bounce off of the walls in the room. Not having a system that bounces light into the shadows makes the light contrast too abrupt, even if the image was later retouched in post-processing.
Unreal 5 render result without Lumen (image credit: José Tijerín)
