“Avengers: Age of Ultron” – Stereo Q&A

by Graham Edwards

Marvel's Avengers: Age Of Ultron  L to R: Hulk (Mark Ruffalo), Captain America (Chris Evans), Iron Man (Robert Downey Jr.), Hawkeye (Jeremy Renner), Black Widow (Scarlett Johansson), and Thor (Chris Hemsworth)  Ph: Film Frame  ©Marvel 2015

When booking your tickets for the latest summer blockbuster, you’ll probably be faced with a choice: 2D or 3D? With Avengers: Age of Ultron, the question is boiled down to its most visceral form. Is it enough just to see Hulk smash? Or do I want to see Hulk smash in stereo!

According to the latest MPAA report, Theatrical Marketing Statistics, nine out of the top ten box office hits in US/Canada in 2014 boasted a 3D theatrical release. The same year saw the global proportion of 3D digital screens increase to 51% (70% in the Asia Pacific region). Pundits continue to debate the pros and cons of 3D, but as long as the major studios continue to pump out big stereo movies, the desire will remain to make the 3D experience as punchy as possible.

Satisfying this desire on Avengers: Age of Ultron were two stereo conversion facilities: Prime Focus World and Stereo-D. In total, Prime Focus World converted 830 shots for the movie, with production running for three months, and the number of team members peaking at 613 across their London, Mumbai and Vancouver offices.

Cinefex spoke to Richard Baker, senior stereo supervisor at Prime Focus World, about the state of the art in stereo conversion, and about the company’s work on Avengers: Age of Ultron.

Marvel's Avengers: Age of Ultron - Hulk

So how about Hulk? He’s a big guy – does that make him a natural subject for 3D?

The issue of Hulk’s size was an interesting one. We obviously wanted to use the stereo to emphasise his scale, and the natural tendency would have been to pump him up in the stereo conversion. But this actually has the opposite effect, and tends to minimise scale. To increase the feeling that Hulk is much bigger than the other characters, we actually flattened him off a little, slightly reducing his internal depth and ensuring that he was never too separated from the background.

Okay, before we get too deep into Avengers: Age of Ultron, let’s fill in the background. First, is the demand for stereo as big now as it was shortly after Avatar brought it back to the mainstream?

There still is a big demand, although the market has now consolidated, with four main service providers in the conversion industry. Marvel is a great example of a studio committed to 3D: all of their films are planned to release in 3D, and they have recently added more films to their slate. Disney, Warner Bros., Fox, Lionsgate … most of the studios have 3D releases in their line-ups.

Have the techniques changed much since those early days?

In the early days of stereo conversion, it was assumed that linear stereo was always preferable – that it was more natural. In fact the industry learned that linear was not always acceptable – that sometimes it just didn’t look right. Also, filmmakers wanted full control over depth, in order to use the stereo as another storytelling device, alongside the grade, edit and sound mix.

This is where stereo conversion comes in. We have developed techniques to incorporate linear and non-linear elements into the same scene, including the generation of virtual stereo camera rigs from our converted scenes and delivering them to VFX to allow them to render CG elements that will slot straight into the conversion.

Editor’s note: The terms ”linear” and “non-linear” refer to the way the stereo offset – in other words, the amount of “3D-ness” – reduces from the foreground to the background of an image. With linear stereo, the offset reduces at a constant rate. With non-linear stereo, the rate of decrease follows a curve, to more comfortably emulate what the human eye naturally perceives.

You use a number of proprietary tools that you’ve developed in-house. What are the advantages of doing this?

When we create tools, it is generally to cater to a specific requirement that off-the-shelf tools can’t handle, or don’t handle well. One of the benefits of maintaining an R&D department and developing tools in-house is that you can create them to be exactly what you want them to be. And sometimes we need do things that no-one else is doing, or has thought to do. We specify, build and rigorously test all our proprietary tools, then roll them out across our global network.

The conversion process itself is constantly evolving also. My background is in VFX, so it always seemed sensible to me to use geometry in the conversion process – for example, with characters’ heads, and with certain environments where it is important to maintain consistency and accuracy of depth across shots. We use cyberscans of the main actors to produce head geometry for all our shows, and we also request lidar for environments that we know would benefit from geometry.

Watch a video about DepthGen, one of Prime Focus World’s proprietory stereo conversion tools:

What kind of background or training do you need to become a stereo artist?

As with VFX, the best stereo supervisors and artists understand and embrace both the creative and the technical aspects of what they are doing. Conversion is very much a VFX process, and we have a range of levels of compositing artists in our stereo teams – seniors, mids and juniors. Being able to really see the subtleties in a stereo shot takes months of training, of looking at stereo, and developing an eye for what works and what doesn’t.

Stereo conversion can provide a great grounding for junior artists who may decide to move into VFX later, and with most of the big blockbuster movies being released in 3D, this training can prove invaluable, not just in learning to recognise good stereo but also in learning about pipelines, colour space, working in Nuke or Fusion and a whole host of other areas.

Watch a video about another of Prime Focus World’s stereo conversion tools – PFLive:

Let’s get back to Avengers: Age of Ultron. How did Prime Focus World get involved in the show?

We were awarded the work on Avengers: Age of Ultron on the back of our stereo conversion work for Marvel on Guardians of the Galaxy. That was a great project to be involved with creatively, and it went on to win an award for “Best 3D Live Action Feature” from the Advanced Imaging Society.

At what stage in the production did you actually start to work on the show?

I was invited out to view an early cut of Avengers: Age of Ultron at the end of 2014. Even though it was all still previs and greenscreen at that point, the edit was already in great shape. I sat with Evan Jacobs, Marvel’s stereographer, and Mike May, Marvel’s stereo producer, to talk through the creative aspects of the conversion and discuss production details. This allowed me to put together my depth summary for the show – an internal guide for our international teams that describes how we intend to approach the stereo, how it will play out across the sequences we are working on, and where we can use the depth to create “3D moments”.

Marvel's Avengers: Age of Ultron - Scarlet Witch

Was any of the film shot with a stereo camera rig?

No, the whole film was stereo converted, although there were some stereo renders – for example, the Iron Man HUD shots. Robert Downey Jr. was shot mono on greenscreen for these sequences, and this footage was subsequently stereo converted. The converted shots were then delivered to Cantina, who comped the stereo HUD elements into the shot. Stereo renders of graphics content and transparencies always look better.

Does it make for more work, doing the stereo conversion in post?

To shoot Avengers: Age of Ultron native would have been nigh-on impossible. There was so much greenscreen content that making depth choices for the live-action characters would have been tantamount to setting the stereo blind, because the backgrounds hadn’t even been created yet. Also, for the VFX houses to work in and render stereo is a big overhead. Adding this to a tight production schedule and hugely complicated creative work would have been an extra hit they didn’t need.

Do you get a superior end result this way?

Ultimately, being able to control the stereo in post is the best way to go. The quality of the stereo result is no longer in question. In fact, it is arguably superior to native, unless native has been through its own post process treatment. Perhaps most importantly, it gives studios and directors the flexibility to make changes to the depth as they see the film come together. With native shooting, you are locked in to the decisions you made on set, so you’d better hope that they were right!

Tell us more about how the stereo “look” of the film was planned.

Stereo decisions are generally made by the show stereo supervisor – Evan Jacobs in this case – in conjunction with the director. For Avengers: Age of Ultron, Evan and I discussed the stereo up-front, making a lot of references to the style and direction that we had set up for Guardians of the Galaxy. Once the brief was clear, we started working up the depth in individual shots. Specific comments or decisions would come up later during client review. As the stereo supervisor across the entire show, Evan was seeing the big picture, and thinking about how the depth was coming together for the whole movie, rather than just for any one sequence.

How closely do you liaise with the VFX vendors during the stereo conversion process?

Our production team has a great rapport with the Marvel production team, which is invaluable on a big, complicated show like this. We also have great relationships with the VFX vendors, which is crucial when we need to harvest VFX elements early. ILM couldn’t have been more helpful. Their incredible one minute-long opening shot for Avengers: Age of Ultron was one of the last renders to come through to us, and they were there 24 hours a day to assist us with breaking it out for conversion.

Marvel's Avengers: Age of Ultron - Church battle

Describe your workflow for stereo converting the film’s big VFX sequences.

The huge battle in the abandoned church towards the end of the movie is a good example of a massive, elements-based VFX sequence. We started working with ILM early in the conversion process to get element passes so that we could start to set up the scene in stereo, even though we knew the animation was likely to change. This benefited Marvel, as it gave them an early temp stereo version of the shot so they could see how it was going to play in the cut, and for DI. As newer versions of the shots came in, we incorporated the changes using our internal tools. This kind of collaboration is crucial to delivering shots like this, with hundreds of VFX layers, within the allocated production timeline.

In terms of the elements pipeline, the VFX work was made available to us in comp scripts containing all the constituent elements and layers that we needed for the conversion. We used our proprietary tool, AssistedBreakout, to render out smaller “minicomp” versions of the huge VFX files, giving us a more manageable script. Without this, the breakout process would have been manual, and taken skilled artists much longer to perform.

Watch a video explaing how Prime Focus World’s AssistedBreakout operates:

The depth generation was handled through our View-D conversion process, using depth mattes, Z-depths (where available), geometry and hand-sculpting to create a depth map which gave us the stereo offset. I reviewed every shot first here in London and delivered any internal notes. Once I was happy, the shot went forward to the client for approval, and any client notes were reviewed over TVIPs, our live stereo review system, with Evan Jacobs in LA and me in London. Once the shots were finaled, they were delivered as 2K log DPX files.

Is it easier to create effective stereo for an element-based VFX shot, rather than live-action straight out of the camera?

Let’s say you had two similar shots: one with digital characters and lots of CG dust, smoke and explosions, and the other shot live-action with practical effects. The benefits of having access to the VFX elements would be most apparent in the cleanliness of the image. The live-action shot would require lots of paint work to fill the occluded areas – which can absolutely be done, and which we have done many, many times – but with VFX elements available we would be able to approach the shot in a different way. And it would be quicker.

On VFX-heavy shows – the majority of the shows that we work on – having the VFX elements allows us to extract Z-depths for the CG characters, and to separate the various layers of smoke, dust, particles and explosions. This reduces the amount of paint required and allows us to work at a higher quality with even more detail and accuracy.

Which shot in Avengers: Age of Ultron gave you the biggest challenge?

I guess if I had to choose the most challenging shot it would be the opening sequence. It was over a minute long. It was a continuous shot. It was one of the last to deliver. ILM had split it into three parts, and we had to split it into eight parts to make it more manageable in the time-frame. It was challenging from an editorial point of view, from a production point of view and artistically too. But we’re used to working with long shots after our work on Gravity!

Which shot do you think gives the most stereo bang for buck?

I think probably the two slow-motion shots that we worked on – the iconic tableau of the Avengers flying across the screen in the opening forest sequence, and the big rotating battle shot in the abandoned church. These are both perfect 3D moments, giving the viewer time to really take a look at the stereo. In the abandoned church shot, you have characters flying in and out of the action, towards the screen, plus there’s a really nice camera move, lots of atmospherics and dust and lasers and lightning … it’s all going on!

Marvel's Avengers: Age Of Ultron  L to R: Hulk (Mark Ruffalo), Captain America (Chris Evans), Thor (Chris Hemsworth), Iron Man (Robert Downey Jr.), Black Widow (Scarlett Johansson), and Hawkeye (Jeremy Renner)  Ph: Film Frame  ©Marvel 2015

You talked about big VFX shots coming in very late in the day. Since stereo conversion is one of the last jobs to be done, how do you cope with the time pressures?

In terms of the VFX, yes, we’re one of the last to touch the film, which means there is huge pressure on the stereo team. By the time we receive turnover, many departments have overrun – through no fault of their own – either because of reshoots, or edit changes, or other delays. It’s an inevitable aspect of the process. We know that the biggest VFX shots are the ones that will deliver last because they are the most complicated. It’s just the way it is, and we anticipate this in our schedules. We know that the last month of a big project like Avengers: Age of Ultron is going to be super-intense!

And yet, working with Marvel, this push to the finish line is also enjoyable. We have weekly calls with the entire post team, and everyone is working towards the common goal of delivering the best quality work and hitting their delivery dates. This openness and transparency is so important. One of Prime Focus World’s great strengths is that we can mobilise our global resources very quickly to ensure that we can handle any situation.

"Marvel's "Ant-Man"

Prime Focus World is continuing its relationship with Marvel on Ant-Man. Does the film’s microscopic hero make it a good subject for stereo conversion?

Ant-Man is going to be a whole new challenge, with its own stereo style driven by the scale considerations of the main character. We’re talking to Marvel a lot about how we develop the stereo language for this show. How does our hero look when he is transformed into his tiny form? Does he feel small in a big environment? Or do you imagine the camera has shrunk with him so that he feels normal-sized? This movie is perfect for 3D, because the stereo can play such a huge part in creating a feel for the show, and we can use it to support the storyline. We’re really looking forward to this!

Finally, are there any new advances in stereo on the horizon?

Deep compositing is really exciting for us. Not all VFX houses are working with deep yet, but companies such as ILM, Double Negative and Weta Digital are. Receiving deep comps from VFX allows us to use the depth data to create the stereo offset, because the occluded areas are present within the deep information. Once we create the left eye/right eye offset, the “missing” information is automatically filled in. The development of our DeepGen tool to take full advantage of deep comps in the conversion process is probably one of the biggest areas of advancement on the tech side for us at the moment.

Also our merger with Double Negative last year was a big step forward for the company, and we’re already seeing the benefits of this relationship in our collaboration on Avengers: Age of Ultron, and on forthcoming projects such as Terminator: Genisys and Ant-Man. The close working relationship of our production teams, the ability to access VFX elements earlier in the process, the time and cost saving benefits to our clients – it’s a big step forward in terms of our efficiency.

Read the full story behind the visual effects of Avengers: Age of Ultron in Cinefex 142. For this latest Marvel spectacular, VFX supervisor Christopher Townsend oversaw a visual effects team that included ILM, Double Negative, Animal Logic, Luma Pictures and Framestore, with special effects supervisor Paul Corbould and the mechanical wizards at Legacy Effects lending practical effects support. Buy your copy of Cinefex 142 now!

Special thanks to Tony Bradley. “Avengers: Age of Ultron” and “Ant-Man” photographs copyright © 2015 by Marvel Entertainment.

3 thoughts on ““Avengers: Age of Ultron” – Stereo Q&A

  1. I go out of my way NOT to see a 3D version of a movie. From the standpoint of the audience it’s an automatic six month delay on the movie’s release because of the conversion process–probably longer. Studios should release it flat when it’s finished and then 6 months later re-release it as a “special 3D” edition to those who care. It would make going to the megaplex that much easier. Seems like a hyped-up, expensive gimmick that they justify by telling everyone how cool it is.

    • You don’t seem to realize the conversion process takes place as the movie is being made, most conversions finish within days of vfx vendors completion of the film.

  2. Well I disagree, I m happy to wait for a quality 3d version to be released, and this shows how much work and collaboration goes into such amazing effects. Well done to all involved.

Comments are closed.