Working Around Limitations

At the opening of the chapter I came right out and admitted that After Effects is not most artists' first choice as a dedicated rotoscoping tool. This is not a generalized complaint or simply a matter of taste: Other programs' masking tools perform useful things that are impossible to do with After Effects 6.5. This section will identify some of those holes and focus on the strategies that can help you make the best of the situation. For example

• There is no way to apply a tracker directly to an After Effects mask, let alone track individual mask points. That doesn't mean there's no way to track in a mask selection, however.

• You can't translate a mask the way you would a layer, by selecting all of the keyframes you want to translate and performing the move; After Effects applies the move to the current keyframe only. The workarounds are similar to those for tracking.

• After Effects lacks the ability to specify whether a feather is applied to the inside or outside of a mask, nor can you vary feather settings on a per vertex basis (Figure 7.11).

  Figure 7.11. All of the motion blur in this shot is left to right, so the ideal mask for the front of this truck would have heavy feathering on the right edge and none on top. Because After Effects lacks per-vertex mask feather settings, however, you have to handle this with two masksnot so bad in this case because the motion is highly predictable.

  

  
  

• Adding points to an animated mask has no adverse effect on other keyframes applying to that mask; the mask retains its shape. Deleting points, however, deforms the other keyframed versions of the mask. Avoid deleting points on a mask animation at all costs. Always start by keyframing the simplest shape, with the fewest points, and work your way up to the most complex.

  

  In General Preferences, there is a preference, on by default, to preserve a constant vertex count when editing masks. Leave this on to prevent inadvertent popping of extra vertices and awkward tweening around missing vertices. But in cases where in-betweening is not an issue (presumably because you're creating a mask on every frame), you can consider unchecking this preference.

  
  

• There is no dedicated morphing tool in After Effects. The elements to do a morph, however, exist in the program.

In the following sections, you'll explore the above points in depth.

Tracking and Translating

If you cannot apply a tracker directly to a mask, even using expressions, what are your alternatives? The next chapter will concentrate on successful tracking, so here I'll focus on simply getting set up.

One workaround that is effective in limited cases is to stabilize the layer, pre-compose it into a composition large enough not to crop the result, and mask the stabilized element; you would then reapply the stabilization by linking Position keyframes to the stabilized Anchor Point either via expressions or a simple copy and paste. This might simplify matters when the element being rotoscoped is itself relatively static but the camera is moving; however, it would not work so well if the element is moving around enough to generate substantial motion blur. Really, it's just an idea to keep in the back of your mind that might be useful once in a great while.

More likely, you will have luck tracking in a mask when articulating its points with the track isn't so important. One example is tracking an element with a solid surface that doesn't change shape. The workaround in this case is to use the masked layer as a track matte and track the transforms of that. The tracked layer would carry the mask with it and then be applied to the alpha of the same layer (Figure 7.12).

Figure 7.12. if you're creating quick holdout masks of items that don't change shape, such as the windows of this car, you can set the masked frame to be a separate layer and apply your track to that, then use the result as an alpha track matte. It's somewhat of a kludge, but it can be helpful (especially when creating garbage mattes, as seen in the next chapter).

This same not-very-pretty workaround can also get you out of a tight spot in which you have dozens of mask keyframes that simply need to be offset, scaled, or rotated. Duplicate the masked layer and set it as an alpha track matte, offsetting the track matte layer instead of each individual mask keyframe. Here's where you're truly out of luck, however: If only one or some subset of the points of your mask needs to be offset, there is simply no way to get that edit to ripple through all instances of that mask point. Features for future versions, perhaps.

It is possible to apply a tracker to a mask point using scripting, but I've never heard of anyone trying to design a script to do it in any useful way. One problem is that the user interface is beyond what is readily possible with a script, because you would want to be able to apply an arbitrary number of tracks to an arbitrary number of points.

Auto-trace

Wouldn't it be great if there were a tool in After Effects that would track masks for you? The good news is there is. The bad news is the cases in which it would be useful are cases in which you should be pulling high-contrast luminance mattes instead. The further bad news is that the process is much slower and more cumbersome than using the techniques from Chapter 6 to create a hi-con matte.

Here's the problem: Auto-trace is a threshold tool that looks either at overall luminance or a specific color channel. You set the threshold (which you can guess reasonably intelligently by sampling pixels using your cursor and the Info palette in Percent 0-100 mode. Next, add a pixel tolerance setting (whole values only), and then let Auto-trace work its magic (Figure 7.13). The result is an arbitrary number of masks, each describing a single area that meets the criteria. Over the course of the shot, some of these masks will reshape, some will disappear, and some new ones will appear.

Figure 7.13. For the most success with Auto-trace, apply it to footage that is already high-contrast, such as this keyed matte, which was created with a 50% threshold and a two-pixel blur. Even this clean a result will chatter over time, however, but if you ever find yourself needing a vector shape based on a high-contrast image or alpha channel, this is a way to get it.

It ends up being a pretty unwieldy result, albeit cool for the motion graphics designers who want to play with those masks. For effects work, however, it's largely pointless compared to trackable mask points, which After Effects lacks.

Masking Motion Blur

Motion blur happens. That's a good thing; it's a natural artifact of seeing on object in motion captured over a short period of time, and on moving footage it reinforces the persistence of vision required to make a series of still frames look like a moving image (when in fact, a moving image is always just a series of stills). But masking motion blur would seem to be something of a nightmare, as its edges defy careful observation.

Well, true, it's not easy by any means, particularly if the masked object is also articulateda bird flying through frame, for example. Remember to carefully evaluate whether you can pull any kind key or hi-con matte on a blurred element to get all of those lovely edges before you get out the masking tools.

You can even evaluate whether the blurred element is really enhancing your scene. (Hopefully, we're not talking about the starring element of your shot.) Because you didn't shoot it on a proper color matte, maybe you'd be better off eliminating the element from your scene.

Otherwise, there is one thing to think about. Masks respect the motion blur settings of the composition; a masked element in motion will have motion blurred edges. That means you have a chance of matching the motion blur with a mask whose contours would fit the shape of the object if it were stationary (Figure 7.14). To make the mask work with the moving element is a matter of getting the composition's Shutter Angle and Shutter Phase settings right. For details on doing that, see Chapter 2, "The Timeline."

Figure 7.14. Don't forget that putting a rotoscope mask in motion allows motion blur to come along for the ride, simplifying a seemingly difficult masking situation, as was shown in Figure 7.11.

This will probably leave you with edge problems that you will have to cleverly conceal, whether by choking, further blurring the edges, forcing them to the background color, and so on.