ILM's Motion Capture Department has produced animation data for robots, amphibious humanoid creatures and humans in Star Wars: Episode I "The Phantom Menace," mummies in "The Mummy", and for Marfalump, the four foot tall, alien star of the Pepsi TV Commercial Spot "Play Acting." This data was used for shots ranging from primary or "hero" performances to complex multi-character shots involving interaction between actors in live-action film footage and CG characters, to entire armies of CG characters in elaborate battle scenes.

Motion capture data was applied to characters with a variety of methods. Some shots used the original data without modification, while others required manipulation of the motion capture performance with traditional keyframe techniques. Many shots relied on the creation of secondary animation with simulation techniques for cloth, flesh and hair, or the integration of motion capture performances with rigid body dynamics.

The large variety of shots required the development of a range of production techniques for directing motion capture shoots, for integrating motion capture performances with action in background plates, and for incorporating motion capture hardware into more conventional film production environments. We have attempted to refine our understanding of the aspects of motion capture performances that affect the realism and dramatic impact of the resulting shot, and we have developed some rough guidelines for deciding when and how to use motion capture. While motion capture has proven to be a useful tool, the current generation of technology and methods suffer from some important limitations; they require controlled environments, elaborate and technically demanding hardware and software, and a high degree of expertise on the part of the director to envision how performances will read in the final shot.

We have found that the qualities of performance that contribute to a believable final shot vary considerably with the nature of the motion. Action shots, with lots of motion are the easiest to direct. The performer's motion is a reliable guide to how the corresponding CG character will read on screen, and the resulting animation can be modified significantly by changing the timing or amplitude of motions without destroying the physical and dramatic plausibility of the original performance.

The success of close-up shots depends more on subtle aspects of the final CG image that are difficult to visualize on set such as precise framing, slight postural changes, profiles, eye-lines, lighting, and shape. We have found that even a rendered version of the motion capture composited over the background plate can be misleading if a low resolution model and simplified lighting is used for visualization. Furthermore animation from these performances is more difficult to modify successfully.

For all but the most generic motions some form of on-set visualization is critical for successfully directing motion capture performances. Our primary technique was synchronizing performances and data capture to background plates on video. Timecode from the video was used to trigger our motion capture system and thus provide known synchronization between our motion capture data and the background plate. This technique was used heavily for both "Star Wars: Episode One" and "The Mummy." In particular a number of battle droid sequences required droids to dodge through a crowd in the film plate or perform carefully timed actions. An elaborately choreographed fight scene in the final sequence of "The Mummy" required the lead actor to interact with 21 computer generated mummies.

The Pepsi Alien project required us to develop a technique to allow the director precise control over the framing of the CG character during a motion capture shoot. The final setup included the creation of parallel sets on adjacent sound stages. A beauty set was used to shoot background plates and block out performances. The motion capture set was setup in a green screen environment. A film camera on the motion capture set was matched to a corresponding camera in the beauty set. This setup allowed the director to see a composite image of the motion capture actor over the background plate while directing performances.

Our work integrating motion capture into existing film and computer graphic production techniques has given us a good overview of the strengths and weaknesses of the current generation of motion capture technology. More importantly it has given us the opportunity to develop some clear ideas about how the technology needs to evolve in order to become useable in less specialized environments and to lower the technical barriers to animators and directors.

Seth Rosenthal is the Supervisor of Industrial Light and Magic's Motion Capture Department. His brief career in special effects includes credits for "Star Wars: Episode One, the Phantom Menace" and "The Mummy." Prior to moving to ILM, he co-managed, along with John Pella, the production of 3D media and data for Microsoft's now tragically defunct Digital Media Center. While at Microsoft he collaborated with the Human Figure Animation Project of Microsoft Research to adapt their motion capture processing and animation system for use in production. He earned a BA in history from Oberlin College.