iClone and the Making of the Mood Indigo Music Video by Jason J. Brown

Behind-the-Scenes

Jason J. Brown is a digital media specialist and 3D artist that produces digital content such as animated graphics for television and video projections for musical theater companies.

He also produces workshops specializing in shooting, editing, and greenscreen. Jason uses animation, robotics and projections to produce work for musical theatres and nightclubs. He has made many YouTube tutorials outlining his digital techniques– particularly for iClone and HitFilm Ultimate.

THE BAND

The Tiki Collective is a group of popular jazz musicians and vocalists who have just released a new album called “Muse“. I was tasked with creating a series of music videos that would complement the South Pacific themes.

The band is led by Jaymz Bee, a well known Canadian radio host and jazz impresario. A mutual friend showed Jaymz some of my earliest green screen work, and since then we have collaborated on numerous video projects. Half of our projects are “straight-up” video documentation of concerts and events. The other half consist of music videos and comedy skits done with green screen and CGI. Many years ago I did my first iClone 4 greenscreen compositing tests with Jaymz’ comedy troupe. (back then we were shooting on DV tape!) Last year we did an amazing 360 music video, called “Guns Kill Guns“. In it, I had the opportunity to both motion capture Jaymz and greenscreen him into a series of 360 worlds based on the movies Scarface and The Matrix.

THE TOOL

There is a huge technological leap from our first DV skits to our last 360 video. It would not be possible to for a small studio to generate dozens and dozens of CGI worlds without a super accessible CGI tool like iClone. While its learning curve has remained shallow, the iClone toolset has evolved exponentially since the 4.0 version. I would not have believed it would grow to support things like face and body motion capture, curve editing, 8K, PBR and other visual effects. iClone also gives the option of exporting animation in 360 video, and in anaglyph stereo. In the future, when I plan new video projects, I can think about delivering in as many formats as possible without too much additional work.

Also, I’m not a modeller, as I much prefer to spend my energy building sets and directing action and animation. So instead of modelling, I hunt my assets from the Reallusion Marketplace — which attracts content developers from all platforms– Google/Trimble 3D Warehouse, TurboSquid and CGTrader. Asset prices range from free to cheap. But, most of the top-tier RL Marketplace developers (Fashion Alley, Warlord, Dicho, etc.) will happily customize or create assets for you.

THE SINGER

The first song from the MUSE album is a performance of Duke Ellington’s “Mood Indigo” performed by Tyra Jutai, a young up and coming jazz singer/songwriter.

Tyra has dazzled audiences on both local and international stages, in Montreal, Toronto, Ottawa, and Amsterdam. She combines the melancholic with the joyful, and humor with earnest sobriety. No one is sure what to expect when she steps up to the microphone, or comes out with a new original song.

THE PROCESS

Using a Panasonic 1080p camera oriented sideways, I started by filming Tyra as she performed several different takes of the entire song. At this stage, no firm decisions had been made about the backgrounds. So I filmed both full body shots, and closeups.

This slideshow requires JavaScript.

I also filmed some “ambient” scenarios such as sitting by a fire, sitting at a bar, etc. In the end I managed to use almost everything we captured.

I used HitFilm 2017 to create masks and mattes of the performances. These mattes were then inserted into the various 3D iClone projects. This method permitted me to continuously refine the CGI scene while only ever compositing the actor once.

I had already built a massive island set with multiple exteriors; during the course of the project I built six additional sets. It was to create the nightclub scene.

To create the nightclub scene, I started with the singer’s performance and then built everything else around her.

For each CGI location, I rendered out the singer’s entire three minute performance. I then put each rendered shot into the video editor on separate tracks and then cut back and forth between the different shots. As the CGI locations were tweaked and re-rendered, the video editor automatically updated the edit with the new footage.

The client asked if it was possible to discreetly feature one of the producers in the video. So by using a few face photos I created an avatar, and positioned him inside the bar scene, motion captured and lit him.

The trombone player was shot a few weeks later at a different studio.

There were two closeup captures that the client wanted to feature — but without the close-ups actually filling the screen. This meant finding ways to re-frame the closeups.

The first concept was the magazine cover. Since the magazine would appear throughout the entire video, I also needed to create a “situation” around them. So I settled on the idea that the Island women were operating a newstand, selling magazines to soldiers.

This slideshow requires JavaScript.

For the second closeup, I decided that the singer would appear behind the windshield of a vehicle on a long drive to her singing gig at the nightclub.

By using an iClone terrain, which had a clearly defined path– I planted tropical trees and flowers. Driving at the slowest reasonable speed for the full length of the song required almost a digital kilometer of landscaping and fog effects!

PopcornFX Fog generating particles are tracked behind the Jeep in order to create right the distance. Other fog particles can be seen following alongside the Jeep.

The iClone TOTAL PLANTS Biosphere Collection was an absolutely crucial tool for planting massive landscapes in all sorts of geographical zones! The variety of plants and trees found in this pack is astonishing.

I also used DEXSOFT Game’s Tiki Island for much of the island’s exterior decor.

For a (distant) future edit of the music video, I rendered out an anaglyph 3D version of the footage. The stereoscopic effect evokes those old Viewmaster reels, but in moving video!

* (red/cyan glasses required)

This slideshow requires JavaScript.

SUMMARY

From a strictly animation standpoint, this video was not complicated to make. There is a minimum of motion capture and character animations, and it was not a 360 video so normal set-building cheats were employed.

Actually, the biggest problem I faced was with the trombone player’s performance. It turns out that the trombone player had totally improvised his performance on the studio recording. So it was nearly impossible for him to replicate his performance for the video capture. In post-production I had to re-time some of his motions to try to match the studio recording. It was a nightmare. In this kind of video you can’t allow any of the animation to detract from the live performance. The digital avatars cannot upstage the singer. So it was necessary to find a way to keep the avatars under-animated and under-lit without them becoming statues or zombies.

In the future, especially now that I’ve upgraded to 4K cameras, I would shoot fewer close ups, since closeups limit the possibilities of the compositing. You want to capture the performer’s entire body at all times. I will also add 4K GoPro cameras into the mix, so that the multiple camera angles can be captured simultaneously. Overall, I found that this general workflow allows the most flexibility in post-production, since you can continue to add details to your shots without constantly re-compositing the actors. You can also create completely new shots that were not planned in advance.

Our next music video with Jaymz Bee and his crew will feature them as angels and zombies. We will be using a special effects makeup artist and a flying-wire harness for the first time. Zombie horror is very well represented in the Reallusion Marketplace; already I have sourced out tonnes of excellent horror props, locations and set-pieces. That video is going to be very crazy and a lot of fun to make!