How ILM created the Hulk

Marvel Studios' cinematic thrill ride continues with its latest action-packed offering, Avengers: Age of Ultron. The action is simply enormous, with each scene delivering new visual treats; building on the visual effects of the 2012 blockbuster.

Joss Whedon returned to write and direct the follow-up as did the ensemble cast of actors playing the superhero Avengers. New to this film is James Spader who voices the sentient, troubled and villainous robot Ultron.

Also returning was visual effects supervisor Chris Townsend, who managed over 10 studios for the last Avengers film and approximately 20 studios this time around, including Double Negative, The Third Floor, Trixter Film; with Industrial Light & Magic contributing a large portion of the shots.


Ben Snow supervised the crews at ILM in San Francisco, Singapore, Vancouver and London, who worked on the studio's 800 shots including the opening and closing battles, shots of Ultron raising a city (literally), and the three main CG characters: Hulk, Iron Man in all his forms, and the new anti-hero Ultron.

Managing Hulk's Bulk

Marvel's 2012 film The Avengers set the Hulk free on the streets of New York City. Artists at Industrial Light & Magic, who transformed actor Mark Ruffalo into the green behemoth, received a VES nomination for their work.

In The Avengers: Age of Ultron, the Hulk appears in 50 per cent more shots than in the earlier film. "Chris [Townsend] and Joss [Whedon] and everyone at Marvel wanted the Hulk to be an ensemble character that they could use anywhere," says ILM animation supervisor Marc Chu.

"In the first film he was very precious, always perfectly framed. Moving forward, they loved the look they got from Winter Soldier, and used the same DP [cinematographer Ben Davis]. So, they shot this movie looser and more realistic.

"As they wanted the Hulk to feel like a natural character, we completely revamped the model in the sculpt and in how we did the muscle rig."

Typically, artists at ILM model the final form for a character's body and then put muscles inside and skin simulations on top. To help control the bulky giant's muscles and flesh for this film, character technical directors Sean Comer and Abs Jahromi created a completely new muscle system.

"With Hulk, for Avengers: Age of Ultron, we built the rig, put the muscle system on top, and the skin on top of that," Marc explains. "So when we animated Hulk, the rig would reverse engineer what the muscles were doing and realistically drive the flesh simulation.

"In the previous film, we had to do corrections to keep Hulk on model, but the new system kept him more on model. For example, if he put his shoulder through a range of motions, it kept the shoulder area from collapsing. We could still correct anything that penetrated, and adjust the jiggle to taste, but it made us six times more efficient."

To devise the multi-layer muscle system, the technical directors worked with a professor of medicine to understand muscle tensioning, had a stunt actor perform calisthenics that the crew photographed, and they also consulted technical papers that described how stretching muscles affect volumes.

"The animators used a joint system as usual," says Ben Snow, ILM's VFX supervisor. "That system drove muscle shapes that deformed using medically derived math.

"The underlying muscle drives a tetrahedral mesh on top – this is similar to what we used to do in the past. Linked to that is a fascia solve, a 2D simulation, which is magnetised to the underlying flesh."

The simulation happens automatically once animators have finished, which saved ILM lots of production time. "Sean [Comer] set up the system so when an animator did a 'make take' pass, the system automatically fired a simulation and rendered a movie file," Ben explains. "Sean would show it to me and I'd send it to the render farm. The next day, we were looking at rendered animation. It was a big win."

Facial evolution

While character technical directors rebuilt Hulk’s muscle system, Lana Lan, the lead face modeller, re-sculpted Hulk's face to more closely resemble actor Mark Ruffalo. "We did some nips and tucks and trimmed him up a bit to make him more rugged and handsome," Marc says.

To provide reference and data for the animators, an ILM team captured the actor's performance on location and at Shepperton Studios in the UK.

"When he's on the ground, we started from motion capture," Marc explains. "When he starts to Hulk out, we have to go keyframe.

"For his facial performance, we had a facial camera that we could use when needed, but a lot of what you see is keyframe animation. We had to go way beyond what a human could do to get that comic-book level of expression."

The art of Ultron

Animators working on Ultron had a more difficult task. ILM created the title character for the film, an eight-foot tall, sophisticated robot voiced by actor James Spader.

On set, the actor wore a head cam and the grey Imocap suit, imprinted with 'targets' for motion capture. "He had an entire getup," says Marc. "A long pole with LEDs for the actors to have a proper eye-line and for the cameramen to frame him. A head cam when he was on set interacting with other characters. He owned wearing the [Imocap] pyjama suit. We also had stunt actors available on set and on our stage here at ILM."

For facial animation, though, Marc relied on his team's ability to replicate James's expressions. "There are cases where facial capture works well, such as doing a photoreal human and a lot of dialogue." Marc says.

"But, for this show I felt a keyframe approach, which is the same approach I used with Davy Jones, would
work well. Animators have a
great skill in breaking down facial performances and translating them to characters."

Before the animators could apply Spader's facial performance to the metal robot face, modellers and riggers had to design a system that would move hard surfaces without bending them.

"Ultron had probably the most elaborate rig we have ever done," Ben says. "It was 10 times the complexity of Optimus Prime. The riggers had to set up a system with plates that slide across each other.

"The cheeks would slide around the eye socket and into a scowl. The lips pushed up under the nose. The chest moved in and out. It was a tricky rig. We shared the rig with Double Negative, much to their horror. We couldn't send the actual rig because everyone uses such different technology, so they had to re-rig it.

"Over the phone they said, 'What the hell? How do we cope with this?' But fortunately they were left to deal the action and we just did the dialogue."

Modelling metal mechs

"Ultron is the hardest thing I’ve ever built here," says digital model supervisor Bruce Holcomb, who has modelled characters and spaceships for Captain America: The Winter Soldier, Star Trek Into Darkness, The Avengers, Iron Man 2, Avatar, Transformers, Terminator Salvation, Pirates of the Caribbean, among other films: his statement carries weight.

"What we had to deal with was a metal version of Davy Jones," he says. "But, Marvel wanted everything mechanically correct. And, Marvel and Joss [Whedon] were nitpicky about capturing James Spader's every nuance.

"His eyebrows, his eyes, his facial tics. The actor has quirky head movements. He walks in a flamboyant manner. But we couldn't have any metal plates bend."

Bruce worked hand in hand with character technical director Abs Jahromi to give Marc the controls and model parts he needed. Bruce envisioned the modelling process would happen in three phases over eight weeks.

But while they had worked from a full set of drawings and data for Iron Man, for Ultron, the modellers had received only two pieces of concept art: Ultron's front from his knees up with his hands curled into fists, and the side of his head.

"It was a constant back and forth process," Bruce says. "It took seven phases, and we kept working on it pretty much right up to the end. We had to create 60 to 70 per cent of the character, which is unusual for a hero character."

Moreover, they had to model a face and body that could move like James Spader. "Every plate in Ultron’s face moves and all the parts move in concert," Bruce says. "His eyes have irises that move. Nothing stretches. He has weird dials in his cheeks. His upper body and scapula is made of metal parts that move like muscles.

"The plates slide and snap back. When we got the model done, they said we needed more mechanics. They wanted a watch within a watch. Somehow we nailed it."

However, as the animators began to replicate James Spader's performance, the model had to change. "Ultron would show up in another sequence and James would do something else," Bruce says. "So the geometry needed to change for the new rig to create certain expressions. They'd animate a shot and his eyebrows would collide with his eyes.

"We had to cut metal plates into smaller plates to get more flexing and keep them from interpenetrating. I didn't tally the number of parts at the end because he was always in flux."

With each new VFX-rich blockbuster, production studios ask for more while shrinking the time VFX studios have to produce something spectacular, but somehow, the superheroes at ILM have succeeded again.

Find out about ILM's work at www.ilm.com

Words: Barbara Robertson

Barbara Robertson is an award- winning freelance writer based near San Francisco. This article was originally published in 3D World issue 196.

Liked this? Read these!

• The VFX secrets of Avengers: Age of Ultron revealed
• Marvel reveals Avengers concept art secrets
• The Cinema 4D secrets of Iron Man's interface