I've said it before, and I'm saying it again: When animating, or articulating, mattes, keep it simple. All of the strategies offered here are for setting up and animating your mattes with as few steps as possible. Not only will this save you time in setup, you'll have fewer elements to fix if something goes wrong. Each new edit you keyframe introduces the possibility for more error, and each new point you add to a mask has to be accounted for on every other frame.
The first question to decide when starting a roto job is whether you prefer to mask in the Composition panel or in the Layer panel. The advantages of the Composition panel are that you can see the layer in the composition and that it updates live as you work. This approach, however, does have several potential pitfalls:
You may have guessed by now that the Layer panel is generally a preferable place to create masks. Tweaking mask points in the context of the entire comp is often useful, but I recommend this only as a secondary approach. If you start to create a mask in the Composition panel and realize you're in trouble, double-click the layer in the Timeline; your mask remains live in the Layer panel, ready for you to continue drawing and editing wherever you left off.
The Layer panel includes a couple of important pull-down menus you should be aware of: View and Target. You select your display mode from the View menu (Figure 7.1), choosing Masks, Anchor Point Path, or any effect you might have applied to this layer. Keep it set to Masks (the default) to edit masks in this panel. If Anchor Point Path is selected, you may be unable to edit your masks. If you choose None or any effect, masks will not be displayed.
Figure 7.1. The View menu specifies what is active in the Layer panel; the Render check box determines whether that particular step is displayed, or the source.
Beside the View menu is a Render check box; toggle it to see the full frame source of your footage and any masks.
The Target menu resides along the bottom bar of the Layer panel (Figure 7.2) for one specific purpose. It enables you to select a target mask that is completely replaced if you draw (or paste in) a new mask. This would seem to be a boon for rotoscoping, allowing you to draw in a new mask at any time, but there's a gotcha: The First Vertex of the mask shape (which you can control) and the direction in which its points are numbered (which you can't) must match those of the target (see the "First Vertex" section).
Figure 7.2. Choose a mask from the Target menu and draw a new mask (or paste in a mask shape). The target mask shape will be replaced; this is predominantly useful with a keyframed Mask Shape.
An invention unique to After Effects among Adobe applications, RotoBezier shapes are effective for creating a mask that changes over time because they have fewer adjustable points than Bezier shapes (discussed in Chapter 3). Fewer controls, theoretically at least, mean less ability to inadvertently create the pops and bubbles (also known as matte boiling) that can send your rotoscoping job in the wrong direction (Figure 7.3).
Figure 7.3. Bezier handles are a bad mix with animation; for each point, three points must be managed for each vertex, and crimping inevitably results when handles cross.
The first vertex of the mask is identifiable as slightly larger than the others. If you're having trouble matching an animation between shapes and need to edit it, context-click a desired vertex and select Set First Vertex from the resulting menu. More on First Vertex in the "Morphing" section of this chapter.
If setup alone is preventing you from using RotoBeziers, you can initially draw a mask using Beziers and convert it to RotoBeziers for animating. Context-click on the mask and choose RotoBezier from the menu to toggle it (Figure 7.4).
Figure 7.4. Many useful options are tucked away in the Mask context menu: Not only can you convert a Bezier shape to RotoBezier, you can lock and hide masks, close a shape, and set the first vertex for use with such features as Smart Mask Interpolation and the Reshape tool.
There's only one problem: RotoBeziers are never accurate during initial setup. To draw a RotoBezier, choose the Pen tool (G key) and check the RotoBezier box in the Tools menu. Pressing G repeatedly cycles through the other three mask editing tools; these do not include the RotoBezier check box, as they are for editing an existing mask, not drawing a new one.
In RotoBezier mode, try drawing a basic mask around the bus in ext_busPassbyUncorrected_HD. For now, don't worry about how well the mask conforms to the edge, just try following this one instruction: Place a point at the center of any curve and on any corner you can perceive (Figure 7.5). As using Beziers probably did when you first used them, this takes practice, so don't sweat the result just yet. The mask won't look right until you adjust tension on the points.
Figure 7.5. Your first pass with RotoBeziers should just hit the key transitions quickly and won't look right when you're done. Place a point at the apex of each curve you find, as well as at each corner. The next step is to tighten or loosen individual points.
The literal key to your success when editing a RotoBezier shape is the Alt/Option key. After you've drawn your final point and closed the mask, with the Pen tool and all mask points still selected, hold down Alt/Option. A double-ended arrow icon appears. This is the Adjust Tension pointer. Dragging it to the left increases tension (counterintuitive if you ask me, but you get used to it). Do so with all mask points active, and they all become corners (Figure 7.6a and b), just as if you collapsed handles at a Bezier vertex. Now drag in the opposite direction, and all of the points act as if their (imaginary) Bezier handles are extended to the limits.
Figures 7.6a and b. Contrast all RotoBezier points at full tension (a) and no tension (b) to get an idea of the possible range, and how to drag and adjust. Generally you will adjust points one or two at a time, however, not all at once.
If the Selection tool (V) is active, use Ctrl+Alt (Cmd+Option) to activate the Adjust Tension pointer.
Although it does help dispel the myth that only Beziers are suitable for shapes with corners, dragging the Adjust Tension pointer with all points selected is not the typical method to edit RotoBeziers. Instead, after laying down the initial points of a RotoBezier mask, deselect it (Control+Shift+A/Cmd+Shift+A) and then Alt-drag (Option-drag) the Pen tool (temporarily enabling Adjust Tension) on each point that requires a tension adjustment.
You can freely add or subtract points as needed by toggling the Pen tool (G key; look for the cursor to change, adding a plus or minus sign beside it). With the Pen tool in Add mode you can also select a point simply by clicking on it, and deselect a point by Shift-clicking on it. This means that you don't have to continually switch tools (Figure 7.7).
Figure 7.7. The Add Vertex tool (one of the Pen tool's modes) will also select an existing point, without creating a new one. Thus the Pen tool lets you keep working without clicking outside the viewer; the G key cycles through its variations.
The hardest thing to accept about RotoBeziers is that the shape is inevitably so wrong when you first lay in your points, and each point you add changes the shape around adjacent points. Now that you're used to the basics of how RotoBeziers work, try drawing a new mask (say, around one window of the bus) this way: When you place a point on a corner, before moving on to the next point, Alt/Option-drag to the left on the vertex. Voila: It's already a corner, and your shape is much closer to final on the first pass (Figure 7.8).
Figure 7.8. Once you develop some facility with RotoBeziers, you can adjust point tension on the fly. Alt-drag (Option-drag) to the left to tighten a corner point; other points are unaffected.
Close-Up: Working Quickly
Here is a summary of tips for working efficiently with masks:
The real advantages of RotoBeziers become evident when you animate them, allowing you to resolve transitions with fewer variables than with Beziers.
Strategies for Complex Shapes
Whether you've been won over to RotoBeziers for creating animated masks or not, the strategies for masking a complex moving shape are largely the same. Here are the main points:
Figure 7.9. One key to working quickly is to intentionally create overlapping, separate mask shapes for different parts of your masked area that you know will move independently, or be revealed and concealed (in this case, because of a simple perspective shift). A shape with 20 or more points will be difficult to manage over time.
Although each new mask by default retains the color of its predecessor, your life will be made easier with overlapping mattes if each one has its own color, and the colors are easy to see against the source. As mentioned earlier, you can enable Cycle Mask Colors in Preferences > User Interface Colors to generate a unique mask color with each new mask. Also in the Timeline panel, select the masked layer and press M to reveal masks, then click on the colored square beside each mask to edit its color.
Block in and refine the first shape, then refine as necessary. Check the result at full speed against a contrasting background, just as you do with color keys. Then move on to the next shape.
In some cases, you can finish the matte this way. In other cases, you'll be left with a few stray holes in your matte here and there that you can go back and clean up with paint strokes. The process for working with paint is detailed in the next section; first, take a look at some of the other strategies at your disposal for masking, to evaluate their relative effectiveness.
Working Around Limitations