At the opening of the chapter I came right out and identified that After Effects is not most artists' first choice as a dedicated rotoscoping tool. This is not a generalized complaint or simply a matter of taste: Other programs' masking tools perform useful functions that are impossible to do with After Effects 7.0. This section will identify some of those holes and focus on the strategies that can help you make the best of the situation. For example
Figure 7.10. All of the motion blur on the truck is left to right, so the ideal mask for the hood would employ heavy feathering on the right edge and none on top; this mask is not feathered and does not match. After Effects lacks per-vertex mask feather settings, so you can either combine multiple masks or animate the mask to match the motion of the truck and enable motion blur, creating the feathering effect automatically (Figure 7.13).
Preferences > General includes a preference, on by default, to preserve a constant vertex count when editing masks. Leaving it on prevents inadvertent popping of added vertices or awkward tweening around deleted vertices. But in cases where in-betweening is not an issue (presumably because you're creating a mask on every frame), you can consider turning this off.
The following sections elaborate upon the above points in depth.
Tracking and Translating
If you cannot apply a tracker directly to a mask, even using expressions, what are your alternatives? The next chapter will concentrate on successful tracking, so here I'll focus on simply getting set up.
One workaround (effective in limited cases) is to stabilize the layer, precompose it, and mask the stabilized element (which is now held in place); the motion is reapplied by linking Position keyframes to the stabilized anchor point (either via expressions or a simple copy and paste). This may simplify matters when the element being rotoscoped is itself relatively static but the camera is moving; however, it will not work so well if the element is moving around enough to generate substantial motion blur or shape changes, or if it doesn't stabilize properly.
More likely, you will have luck tracking a mask when articulating it isn't so important; for example, an element with a solid surface that doesn't change shape. The workaround in this case is to use the masked layer as a track matte and track the transforms of the track matte layer, which carries the mask with it (Figure 7.11).
Figure 7.11. The green holdout area is a corner-pinned mask to which a track can be applied (and a corner-pin track can even track the deformation as it moves across frame). The same idea works in any case where a simple mask shape can be tracked without need to deform drastically over the course of the shot.
This same not-very-pretty workaround can also get you out of a tight spot. Suppose you have dozens of mask keyframes that simply need to be offset, scaled, or rotated. Duplicate the masked layer and set it as an alpha track matte, translating the entire track matte layer instead of each individual mask keyframe. If, however, only one (or some subset) of the points of your mask needs to be offset, there is simply no way to ripple that edit through all instances of that mask point. Features for future versions, perhaps.
Wouldn't it be great if there was a tool in After Effects that would track masks for you? The good news is there is. The even better news is that it has been further optimized for version 7.0. The bad news (you knew this was coming) is the cases in which it is useful are cases in which you should be creating high-contrast luminance mattes instead. The even worse news is that the process is much slower and more cumbersome than using the techniques from Chapter 6 to create that hi-con matte.
Here's the problem: Auto-trace is a threshold tool that looks either at overall luminance or a specific color channel. You set the threshold (which you can guess reasonably intelligently by sampling pixels using your cursor and the Info palette) in Percent 0-100 mode. Next, add a pixel tolerance setting (whole values only), and then let Layer > Auto-trace work its magic (Figure 7.12). The result is an arbitrary number of masks, each describing a single area that meets the criteria. Over the course of the shot, some of these masks will reshape, some will disappear, and some new ones will appear.
Figure 7.12. Auto-trace is applied to the alpha channel of a matte created for the previous chapter. Over two dozen masks appear to outline and hold out areas of detail, but clearly this amount of detail presents a challenge; more useful results can be attained using simpler, higher-contrast source images.
It ends up being a rather unwieldy result for precise work, albeit cool for the motion graphics designers who want to play with those masks. For effects work, however, it's largely pointless compared to trackable mask points, which would be fabulous if only After Effects had them.
Masking Motion Blur
Motion blur happens. That's a good thing; it's a natural artifact of seeing an object in motion captured over a short period of time, and on moving footage it reinforces the persistence of vision required to make a series of still frames look like a moving image (when in fact, a "moving image" is always just an illusion created by a series of still images). Masking motion blur would seem to be something of a nightmare, as its edges defy careful observation.
True, it's not easy by any means, particularly if the mask is also articulateda rotoscope of a bird flying through frame, for example. Remember to carefully evaluate whether you can pull any kind of key or hi-con matte on a blurred element to get all of those lovely edges before you get out the masking tools.
You can even evaluate whether the blurred element is really enhancing your scene. (Hopefully, we're not talking about the starring element of your shot, but that happens all the time, too.) Consider whether you'd be better off eliminating the element from your scene.
Otherwise, there is one thing to think about. Masks respect the motion blur settings of the composition; a masked element in motion will have motion blurred edges. That means you have a chance of matching the motion blur with a mask whose contours would fit the shape of the object if it were stationary (Figure 7.13). To make the mask work with the moving element is a matter of getting the composition's Shutter Angle and Shutter Phase settings to match. (For details, see Chapter 2, "The Timeline.")
Figure 7.13. Putting a rotoscope mask in motion via keyframes allows motion blur to come along for the ride, simplifying the seemingly challenging masking situation shown in Figure 7.10.
This will probably leave you with edge problems that you must work to cleverly conceal, whether by choking, further blurring the edges, forcing them to the background color, and so on.