Dec 6, 2017

Neill Blomkamp's latest film stars a real-time digital human

The last time Cartoon Brew covered Neill Blomkamp's experimental film production company Oats Studios was in June. Since then, Oats has released several short films and experimental works, including some produced with the Unity game engine.

The most recent collaboration between Oats and Unity, a short film called Adam: Episode 3, expanded on what Blomkamp's team had already done with the real-time engine, introducing digital humans as the two main characters.

At SIGGRAPH Asia in Bangkok, where Adam: Episode 3 was presented, Cartoon Brew again interviewed Chris Harvey, visual effects supervisor at Oats, to explore the specific He explored the challenges of rendering digital humans in real time.

In brief, Blomkamp established Oats Studios in Vancouver after having already made several studio feature films with extensive use of complex visual effects ("District 9," "Elysium," "Chappie"), and Oats goes back to the director's roots in short films, where he has been working on a number of films that are now in the process of being released in the US, designed to experiment with content and explore new ways to connect with audiences. Short films were shared online as well as direct assets from the work.

Despite being experimental, the shorts followed an almost typical production lifecycle from pre-production to post-production; CG and visual effects were an important aspect of most of the shorts, which is a result of Oats' use of motion capture, photogrammetry, complex modeling and rigging, and various rendering techniques were employed in the pipeline.

And at one point, Unity Technologies (makers of the Unity game engine) approached Blomkamp with the idea of applying the experimental aspects of "Oats" to a short film that would be rendered in real time. Unity had already done so in 2016 with its own technology demo called "Adam," which was effectively a short film in its own right; Unity's intention was to highlight the game engine's physically plausible nature and its capabilities as a filmmaking tool.

Oats took over Adam's assets and developed additional assets to create Adam: Episode 2 in Unity. This was a completely new endeavor for us, and a learning curve," Oats said. "One of the big questions was, as a traditional filmmaking company, how do we adapt to using something like Unity? In fact, the transition to something like this was very seamless."

Adam: Episode 2 was actually created in much the same way as Oates' other short films. Assets were modeled, rigged, textured, and animated with conventional CG tools. Action was also captured with performance capture, including facial capture; Unity's timeline sequencing tools allowed for scene management and editing.

For Harvey, the advantages of working in real time were clear. Real-time does not require a render farm, so it can be iterated over and over again. Literally, the film always existed live. Literally, the film always existed live. We could turn the camera, move the lights, add objects. We could do whatever we wanted, and it was always moving.

"At one point," Harvey added, "we reworked [the female character's] mask. I think we redeveloped [the female character's] mask. Because at some point the mask covered so much of her face that you couldn't see her emotions. I wanted to take that away and make the whole thing glassy. And I said, 'We really need to put this in. We need to see how it starts to look.' I put it in the shoot, and then a week later you get to see it. But a few hours later I was at another artist's desk looking at where to put the stones. My mind went blank." It was like, "Oh, it's always there live."

The real-time production aspect was an obvious advantage to the filmmaking process for Oates. Adam: Episode 3 attempted to do just that. The human animation in the short film was informed by body and facial capture, but the main approach to making it photorealistic was to scan the actors for reference and use Unity shaders to create sub-surface scattering (light penetrates the skin surface and phenomenon) that interacts and bounces back.

Subsurface scattering in traditional offline renderers is key to the more believable digital humans produced in Hollywood blockbusters. While it has certainly been tackled before in several real-time projects, including other game engines, this was the first time it had been attempted on such a large scale in Unity. In a sense, according to Harvey, Oates completely ignored the fact that they were using a game engine and tried to solve the digital human problem as if they were making a generic movie. For example, they used the Alembic format (common in many visual effects projects) to stream cloth and facial animation.

"When I joined this project," Harvey said, "I thought, 'How can I make this better? I didn't think, 'How can we make this better? Therefore, I am going to approach it the same way."

Still, there were limits to relying on a real-time engine for something so complex. Therefore, Oates, who wanted the film to run at 30 frames per second in real time, had to make some concessions. He said, "We could have set it up to render one frame per second, but we wanted to render 30 frames per second. By imposing an artificial limit on ourselves, we also set a technical limit within the engine." Our faces have to be at a certain resolution if we want them to stream at that speed. The cloth also only has a certain resolution. Even if we had the ability to go to a higher resolution, there was a technical barrier that we couldn't keep up at 30 frames per second."

Behind Oates Studios may be the influence of figures like Blomkamp and the expertise and experience of many CG and visual effects artists like Harvey, but the real-time films they are currently producing also attest to the fact that a new kind of filmmaking is possible Unity is free for beginners, students, and hobbyists, and Adam assets are available for download; the version of Unity used by the Oats is the same version available worldwide (in fact, the Oats used an older version than the current release )

Harvey recalls that Blomkamp himself was stunned by the filmmaking tools that are now commonplace compared to what he had when he entered the industry. Neil used to talk about this a lot."