F is for Frames

by Graham Edwards

Cinefex - "F" is for "Frames"In the VFX ABC, the letter “F” stands for “Frames”.

In the June 1926 edition of American Cinematographer, pioneering cinematographer Carl Gregory wrote an article entitled Trick Photography Methods Summarized. In it, he divided visual effects into fifteen separate categories.

Here’s a condensed version of Gregory’s list:

  1. “Straight cinematography – a series of frames taken at the approximate speed of sixteen exposures per second”
  2. “Slow motion – in which the taking rate is considerably increased”
  3. “Time condensation – decreasing the taking speed to such an extent that movements which take place slowly appear to occur in a few seconds”
  4. “Trick crank or one picture turn”
  5. “Reverse camera – the showing of pictures in reverse order”
  6. “Masks or mattes – enhancing the illusion of scenes observed through a keyhole, telescope or other familiar orifice”
  7. “Stop camera and substitute – one of the oldest and most familiar of trick devices”
  8. “Fade and dissolve – similar to stop camera, but a gradual instead of abrupt change”
  9. “Double/multiple exposure – for dual roles, visions and ghostly”
  10. “Glass work – a variety of simultaneous double exposure”
  11. “Simultaneous double exposure by means of mirrors and prisms”
  12. “Double printing – making a composite negative by duping from two or more specially prepared positives and masking devices, or in making a special positive from two or more negatives and then duping the result”
  13. “Traveling matte – figures in action may be superimposed against any background without being necessary to build any sets at all”
  14. “Projection printing”
  15. “Other mechanical devices too numerous to even attempt their listing”

Gregory’s list – fascinating to visual effects historians for any number of reasons – splits neatly down the middle. Items 1,2,3,4,5,7 and 8 describe methods that are dependent upon the movement of film through the camera. The rest (with the exception of the catch-all 15) describe a range of compositing techniques.

In other words, half the tricks in the early VFX artist’s magic box were concerned with frame rate.

Carl Louis Gregory, Yellowstone, 1914

Carl Louis Gregory, Yellowstone, 1914 – image by Frank Mt Pleasant Library of Special Collections and Archives

Frames On Film

At this point, I should probably explain the antiquated moviemaking system that prevailed before digital cameras appeared on the scene (and which many filmmakers still favour). Once upon a time, all films were made using, well, film – clear strips of celluloid impregnated with photosensitive chemicals. Sprockets in the camera engaged with regularly-spaced holes in the celluloid, moving the film along in a series of jerks.

After each jerk, the film came to a brief halt. A shutter (mechanically linked to the jerking mechanism) opened, allowing light from the camera lens to fall on a designated rectangle of film. After a predetermined time, the shutter closed, and the film jerked on.

At the time Gregory was writing, film moved through the camera at around 16 jerks per second. When the talkies came along, the demands of sound recording upped the jerk-rate to 24 per second; this has remained the industry standard ever since.

Frames In Early VFX

Carl Gregory’s focus on frame rates is entirely consistent with the thinking of the day. In 1926, moving pictures were an emerging art form, with much attention being paid to the mechanics of their production (some might say nothing’s changed). It was only natural for imaginative cinematographers like Gregory to experiment with their new-fangled contraptions.

I can hear the conversation now:

Look, sixteen jerks a second creates the perfect illusion of fluid movement.

Right, but see what happens if I crank at thirty-two jerks a second, and project at sixteen. Look! Everyone’s walking in slow motion!

Neat! What happens if you crank at just eight jerks?

Haha! Now they’re all jittering around like clockwork toys!

Ooh, check this out. If I crank just one frame, then wait a while, then crank another, I can speed up the movement of this blooming flower, thus shedding light on a natural process previously unobservable by the human eye.

I can do a similar thing to make this model dinosaur trample a caveman!

What should we call this technique? How about “stop motion”?

Nah. “Stop motion” is when you stop the camera, substitute one thing in the picture for another, then start cranking again.

Like that film you made when the actor turned into an umbrella stand?

Right. Let’s call it “trick crank” instead.

Yeah. That’s much more likely to catch on.

And so on.

Gregory’s list illustrates the way visual effects techniques are dependent upon – and indeed driven by – the technology of the day. Back in 1926, that technology was dominated by the jerky passage of a celluloid frame through a mechanical camera movement.

But that was ninety years ago. How have things changed? Are visual effects artists still fascinated by frame rates?

Frames Today

Frames are still the building blocks of cinema. But they don’t hold quite the same sway as they used to. In 1926, an animator might have brought a dinosaur to life using trick crank animation, painstakingly composing each individual frame by hand. Modern dinosaur animators are more likely to think in terms of keyframes – critical poses that might be one, two, four or more actual frames apart – and leave their software to calculate all the intervening frames (inbetweens).

Alternatively, animators might derive the movement of an object based on a spline – or mathematical trajectory. In this case, all they’re concerned with is how long an object (or part of an object) takes to move from point A to point B (and most likely on to points C, D and E), together with its attitude and acceleration as it departs one point and approaches the next. In this case, until render time, frames are essentially irrelevant.

The same applies to sims. For example, an FX artist might use a fluid simulation to generate a tsunami. Frames don’t matter much until it’s time to output the scene, any more than a live action camera operator would pay them much heed when pointing his lens at a real-life tidal wave (let’s face it, he’d more likely be planning his escape route once he’d captured the catastrophe on film).

All the same, even in this digital age, movies are still captured, rendered and projected as a sequence of discrete frames. Ultimately, each individual frame is a vital part of the whole. What’s more, given the tools now available to the visual effects artist, each individual frame can be massaged to the nth degree.

In other words, frames still count.

More Frames

Whether you’re a visual effects artist, a film aficionado, or just an eager fan, you’ll probably know about high frame rates. Put simply, HFR means more jerks per second (not that a digital camera actually jerks, but you get the picture). Like it or not, Peter Jackson’s The Hobbit trilogy has unleashed 48fps upon the world. With James Cameron poised to deliver Avatar 2, 3 and 4 at 60fps, it looks like high frames rates aren’t going away any time soon.

For the visual effects artist, having more frames creates more work. It also means you get to work with a broader palette, providing more opportunities to craft the subtleties of a moving image to a degree not previously possible.

To illustrate, here’s Weta Digital animation supervisor David Clayton, talking about animating Gollum at 48fps on The Hobbit: An Unexpected Journey*:

If we were to animate a blink at 24 frames per second, we’d have two frames down, maybe hold the blink for one, and then relax the blink off over three frames. At 48 frames per second, that would take four frames down, and then maybe hold for two, and then maybe six or seven frames back up. So something that used to take five frames, now takes ten, which means we can break down any movement into smaller, more detailed parts.

Goblin King animation rig - The Hobbit - Weta Digital

With “The Hobbit: An Unexpected Journey”, the high frame rate allowed Weta Digital to introduce extra subtleties into the performances of their digital characters, including the goblin king, performed by Barry Humphries.

Is this broader palette always better? Of course not. But it exists, leading us to the inescapable conclusion that frame rate – along with all its associated advantages and disadvantages – is no longer an imposed restriction but an artistic choice.

It’s no coincidence that one of the earliest proponents of high frame rates was a VFX professional. During the 1980s, Douglas Trumbull developed a 60fps system called Showscan. While Showscan failed to reach the theatrical market, it found a home in theme park rides, and paved the way for the future developments that are beginning to crystallise around us.

No Frames

What if you could get rid of frames altogether? Is such a thing possible?

Frameless image capture records moving images not as a series of frames, but as a continuous stream of information. Light passes through a lens to fall on an array of sensels (as with any digital camera). But, instead of recording the data as a sequence of individual frames, the camera software records the output of each sensel as a continuous waveform. Thus the sample rate – or frame rate – of the camera becomes irrelevant.

The concept of frameless image capture is presented here in Frameless, time domain continuous image capture by Henry Gordon Dietz, Univ. of Kentucky (United States).

Then there’s the idea of adaptive frameless rendering, as presented by Ben Watson, David Luebke, Abhinav Dayal, and Cliff Woolley at the 2005 Eurographics Symposium on Rendering. Instead of slavishly updating motion picture images one frame at a time, this process continuously samples a scene and prioritises what gets redrawn according to a range of criteria. Typically, a area of the scene containing fast motion will be redrawn more frequently than one in which the scene elements are static. In theory, it’s a little like a video compression algorithm. In practice, it’s a step away from the tyranny of the individual movie frame.

This video explains how adaptive frameless rendering works:

The above concepts are yet to gain a foothold in the film industry at large. But their very emergence is evidence that fundamental change may be on the horizon. Which raises an interesting question:

Are the days of the frame numbered?

Conclusion

I’ve barely scratched the surface of frames. If you want to dig deeper (and why wouldn’t you?) there’s no better place to start than this video, in which a panel comprising Bruce Jacobs, Mark Schubin, Larry Thorpe and – of course – Douglas Trumbull embark on The Great Frame Rate Debate (Part 1):

For links to the rest of this debate, visit the PBS Quality Group YouTube channel.

I’ll leave it to Carl Gregory to round things off. It’s impossible to read his 1926 list of the fifteen visual effects methods without wondering what an equivalent list might look like today. Would it feature physics simulations and 3D geometry? What about motion capture, photogrammetry, digital matte painting or rotoscoping?

Whatever might be on that list (we’ll have to leave compiling it for another day), Gregory makes one statement that’s as true today as it was in 1926:

The trick photographer leaves no stone unturned in seeking to produce the desired effect and any device which lends itself to his use is considered his legitimate ally.

Below is an online version of Carl Gregory’s American Cinematographer article from the Media History Digital Library (unfortunately, Gregory’s introduction is on page 9, which didn’t get scanned, but if you zip forward to page 16 you’ll be able to read the bulk of the article without any problem):

*Quote taken from There and Back Again by Jody Duncan, Cinefex #132.
Image from
The Hobbit: An Unexpected Journey copyright © 2012 by Warner Bros Pictures. All rights reserved.