Back in the day, if you wanted to go beyond the rather pedestrian Color Difference Key in After Effects you needed a third-party keyer, and for a few years there were two choices: Ultimatte for After Effects (now AdvantEdge, Ultimate Corporation) and Primatte Keyer (Red Giant Software, formerly Pinnacle Systems, and before that, Puffin Designs).
When I label Color Difference Key "pedestrian," I speak as someone who matted hundreds of shots with it, back before there were alternatives in After Effects. Its methodology is fundamental and can be replicated using basic channel math and Levels controls. It provides the exact methodology that was used for early digital composites (in such blockbuster films as Terminator 2), which itself is the digital equivalent of optical compositing (used in Hollywood classics from The Ten Commandments right through the original Star Wars trilogy).
Debuting with version 6.0 Professional was Keylight, a third well-known software keyer popular among Shake users. It remains part of After Effects 7.0 Professional (but as a separate installation, so make certain that you have it). The other keyers have by no means disappeared, and I personally remain a fan of Primatte, which offers a methodology very different from that of Keylight. Working on The Adventures of Shark Boy and Lava Girl in 3D after the previous edition of this book was released, I decided to use Keylight for all of my shots, however, and I never felt compelled to revert to Primatte. Because Keylight is what you have with After Effects Professional, let's focus on that.
Keylight is useful in many keying situations, not just studio-created blue- or green-screen shots. For example, you can use Keylight for removal of a murky blue sky (Figure 6.11). You wouldn't use Keylight to pull a luminance key, however, or when simply trying to isolate a certain color range within the shot.
Figure 6.11. In one pass, with no adjustments other than selecting the sky color, Keylight does an effective job of knocking out the background even around the motion-blurred bird in flight. The sky is nowhere near a perfect blue screen in this case, other than its uniformity: Blue saturation is around 50% and only about double the amount of red.
Keylight is most typically employed when the plate was shot against a uniform, saturated, primary color background, and when preserving edge detail is of utmost importancein other words, most typically (but not exclusively) when working with footage that was specifically shot to be keyed, against a screen of blue or green.
Generalized Keying Methodology
Figure 6.12 shows a process tree that outlines the steps detailed below (also included on the book's disc as keyingFlowchart.pdf). No two complex shots are the same, but something like this approach should help you pull a good matte, regardless of the tools used. The next section reprises these steps, specifically using Keylight.
Figure 6.12. This chart summarizes steps that can yield a successful color key. It is also included as a PDF on the disc.
Just in case I'm not yet succeeding in beating you over the head with it, my overall message is: Break down plate footage into separate passes when keying. At the very least, a hard "core" matte to fill holes and an overlaid edge matte to recover detail will often improve radically on a single pass. Extra passes are then applied via masked layers to specific problem areas.
The basic steps are
Figure 6.13. The quick-and-dirty garbage matte is your friend for eliminating unwanted parts of the stage.
Figure 6.14. A quick diagnosis of Keylight's first pass at this shot: The shadows must be keyed out because they extend outside the garbage matte area (and right off the set). This in turn has an effect on fine detail such as motion blur and hair, so these areas (hands, hair) may require their own holdout mattes.
If things look noisy and chaotic in the alpha channel or the edges are clamped and chewy, you can
You must also watch out for, and consider rotoscoping, foreground features that can threaten an effective key, such as
Figure 6.15. This is the step too many artists aren't willing to take soon enough. It may seem tedious and cumbersome to create individual holdout mattes for hair, heavy motion blur, and other such detail, but you could waste hours trying to eliminate the shadows while retaining the detail; a quick roto-matte can often be keyframed in 10 or 20 minutes on an average-length shot.
Close-Up: Holdout Mattes
A holdout matte is an area of an image that is isolated for its own separate treatment. An effective key often involves the combination of two holdout mattes, one for the edge and another for the core.
The question then arises as to how these two mattes should be combined; with this method, it should be a simple matter either of overlapping the two layers or precomposing them so that they make a single alpha channel to be applied as a track matte.
In other compositing programs such as Shake, it's not unusual to employ a third holdout, for the background, but in After Effects this is typically avoided and is hopefully unnecessary. The difficulty is that this third matte must be subtracted, rather than added, and transparency (alpha) information is always combined additively in After Effects.
You can convert all of the alpha channels to RGB (using Shift Channels) and apply the Darken blending mode to the background holdout to subtract, but it is frankly a hassle in After Effects, useful only if no other method of holding out the background proves effective. I have seen ex-Shake users drive themselves completely batty working this way.
This basic approach should work with any tool set. The real underlying point is that the attempt to key a plate in a single pass is typically too lazy. Ironically, it often results in much more work, overall, as you struggle to make all areas of your shot accommodate unified, one-size-fits-all settings.
The version of Keylight that ships with 7.0 is new, although the changes in version 1.1v1 of Keylight are relatively subtle.
The most significant of the Keylight parameters remains the Screen Colour selection (Keylight reveals its UK origins at Framestore CFC with that u). Choose the color to be keyed wisely, and you have a shot at a one-pass result. Otherwise, you may fight an uphill battle the rest of the way. More about how this selection works is found in "The Inner Workings of Keylight" section.
You can follow along with the steps using the blueScrn_mcv_HD.mov footage included in this chapter's Pixel Corps folder on the book's disc. Completed versions of several of the examples from this chapter are contained in the 06_colorKey.aep project.
In the best-case scenario, after creating any basic garbage mattes as usual, you will
Figures 6.16a, b, and c. Even when you use well-shot, high-definition source (a), it is imperative to get the Screen Colour setting right if you want to preserve all of the transparent detail in this shot. Choosing a darker background color (b) creates a more solid initial background, but a lighter color selection (c) preserves far more detail. (Source footage courtesy Pixel Corps.)
You can take a snapshot of each attempt (Shift+F5 through F8 to take four individual snapshots; hold down F5 through F8 to see each of them), but the method described holds each pass in the Undo buffer, allowing you to recover the best one.
Close-Up: The Eyedropper and the Info Panel
When you sample a pixel in After Effects, either using an eyedropper, such as the Screen Colour selector in Keylight, or simply by moving your cursor around the viewer and looking at the values in the Info panel, you are seeing the exact value of the exact pixel underneath the selection point of the cursor (the lower-left corner of the eyedropper, the upper-left corner of the pointer). But as with just about everything, there are exceptions.
If you hold the Ctrl/Cmd modifier key, the sample area increases and you get the average of the modified area. By default, this is 5 x 5 pixels, and hackers will even change it by editing the Adobe After Effects 7.0 Prefs file (searching for "Eye Dropper Sample Size With Modifier").
More significantly, in version 7.0 these sampling tools no longer necessarily give you the visible state of the pixel. As in Photoshop CS2, the actual state of the pixel is sampled, which means that with a semi-transparent layer, the value will be sampled as if the layer was opaque (straight transparency values).
Thus, if you sample any pixels in your matted plate, and they are not completely transparent, you get the color values the layer would have with its alpha channel turned off.
Now, as needed, seek out areas that want refining:
Figure 6.17. Sampling a background pixel near the center of the image yields a good initial key, as seen in the Status matte. The black areas are already completely transparent, the white areas opaque, and the gray pixels constitute the areas of focus; they are the semi-transparent regions.
Figure 6.18. This initial result, with a Screen Balance of 95, would be fine but for the ugly black matte line that appears along the arm. Better in this case to stick with the default green-screen setting of 50.
Figures 6.19a and b. Screen Gain is raised to remove the gray in the background. It's tempting to go with a setting of around 120 to remove every last gray speck from the background (a), but a much looser setting (around 105) that leaves some speckling, particularly away from the foreground figure (b), is much preferable for maintaining edge detail. The noise in the corners can be eliminated either via a garbage matte or by raising Clip Black.
This footage is well shot, and although the footage includes flowing hair (a challenge), it does not include the floor, which is usually much more of a challenge.
In this ideal scenario, your matte is complete without the need to go beyond these first four controls (Screen Colour, Screen Gain, Screen Balance, and Despill Bias), which comprise the heart of Keylight.
However, a glance at the current alpha channel reveals a problem. The yellow stripes in the shirt are not fully part of the foreground matte (Figure 6.20) because yellow and green are so close to one another on the color wheel.
Figure 6.20. The yellow stripes on the shirt are not easily distinguished from the green matte.
Also included on the disc is a shot of this same model, wearing the same costume, but taken with a blue-screen background. Try an initial key on that plate (bluScrn_mcu_HD.mov, located on the disc's Pixel Corps folder) and you'll notice that the stripes in the shirt cause no problemyellow and blue are virtual oppositesbut other issues arise with the rim lighting in the hair.
This is the point at which you make the decision to separate out one (or even several) core mattes. Rather than ruining the nice edge detail by trying to get the yellow stripes on the same pass, you now create a separate core matte to take care of the shirt.
Keylight includes its own controls for adding holdout areas to mattes via the Inside Mask and Outside Mask controls. You draw a mask, set it to None, and then select it under Inside Mask or Outside Mask. This works for garbage mattes, but it's not a substitute for duplicating the layer to combine multiple keys.
Figure 6.21. The formerly gray pixels in the foreground are now green in Status view. These are the pixels that have been changed by clipping the matte. The spill suppression that occurred as part of the initial keying operation no longer applies to these pixels.
Figure 6.22. In a perfect world, you could create a matte this good without the need for the holdout matte. But you and I don't live in that world, so you may as well be grateful this shot only requires one mask.
Now what? The mask needs animating, and the discoloration to the shirt will have to be dealt with (see the "Color Spill" section). But without too much pain you have a mask that can be used to place this model and her long flowing hair over whatever background you choose.
Understanding and Optimizing Keylight