Articulated Mattes | Adobe After Effects 6.5 Studio Techniques

I've said it before, and I'm saying it again here: When animating, or articulating, mattes, keep it simple. All of the strategies offered here are for setting up and animating your mattes with as few steps as possible. Not only will this save you time in setup, you'll have fewer elements to fix if something goes wrong. Each new edit you keyframe introduces the possibility for more error. Focus on staying simple.

The first question to decide when starting a roto job is whether you want to mask in the Composition window or in the Layer window. The advantages of the Composition window are that you can see the layer in the composition and that it updates live as you work. This approach, however, does have several potential pitfalls:

On a large scene you may have to wait for the screen to redraw for each point you draw.
If the layer you are masking has been transformed, particularly rotated in 3D, you may find it difficult to edit a mask cleanly.
You may need to select and lock all of the other layers to be sure that you're editing the correct layer, particularly if you're creating more than one mask.
As you create the mask, the masked areas disappear, which can make it impossible to draw complex shapes.
You lose some of the useful selection tools that exist only in the Layer window.

I could go on, but by now, you have probably guessed that opening the Layer window is generally preferable for creating masks. Tweaking mask points in the Composition window is often useful, but I recommended this only as a secondary approach. If you start to create a mask in the Composition window and realize you're in trouble, you can go to the timeline and double-click the layer; your mask will remain live in the Layer window, ready for you to continue drawing and editing wherever you left off.

The Layer window includes a couple of important pull-down menus you should be aware of: View and Target. You select your display mode from the View menu (Figure 7.1), choosing Masks, Anchor Point Path, or any effect you might have applied to this layer. Keep it set to Masks (the default) to be able to edit masks in this window. If Anchor Point Path is selected, you may be unable to edit your masks. If you choose None or any effect, your masks will disappear altogether.

Figure 7.1. The View menu specifies what is active in the Layer window.

Beside the View menu is a Render check box; with a mask selected, disable it if you want to see the full frame source of your footage without disabling a mask (Figure 7.2).

Figure 7.2. The Render check box, if unchecked, disables whatever is selected in the View menu.

The Target menu resides along the bottom bar of the Layer window (Figure 7.3) and does one thing. It enables you to select a target mask to replace if you draw (or paste) in a new mask. This would seem to be a boon for rotoscoping, allowing you to draw in a new mask at any time, but there's a gotcha: The first vertex of the mask shape (which you can control) and the direction in which its points are numbered (which you can't) must match the target's.

Figure 7.3. Choose a mask from the Target menu and start drawing a new mask (or paste in a mask shape). The target mask shape will be replaced.

RotoBeziers

An invention unique to After Effects among Adobe applications, RotoBezier shapes are great for creating a mask that has to change over time because they are much simpler than Bezier shapes (discussed in Chapter 3). Fewer controls, theoretically at least, mean less ability to inadvertently create the pops and bubbles (also known as boiling mattes) that can send your rotoscoping job in the wrong direction (Figure 7.4).

Figure 7.4. With moving footage, Bezier shapes are prone to crimping; managing the length of the handles can lead to error.

The first vertex of the mask is slightly larger than all of the rest. If you're having trouble matching an animation between shapes and need to specify it, context-click on the mask in the Layer window and select Set First Vertex from the resulting menu. More on First Vertex in the Morphing section of this chapter.

There's only one problem: RotoBeziers can look really, really wrong while you're in the midst of setting them up. If this is a problem, you can even try initially setting up your masks as Beziers and then switch them to RotoBeziers for animating. To use RotoBeziers from the start, choose the Pen tool (G key selects it, pressing G repeatedly toggles through the associated variations) and check the RotoBezier box that appears in the Tools menu. To switch a mask shape from Bezier to RotoBezier after, or even while, creating it, context-click on the shape and look for the RotoBezier toggle halfway down the list (Figure 7.5).

Figure 7.5. There are lots of useful options tucked away in the Mask context menu, including the ability to lock and hide other masks, to close the shape, and to set the first vertex for use with such features as Smart Mask Interpolation and the Reshape tool.

In RotoBezier mode, try drawing a mask around an object. If possible, choose something that has a combination of curved edges and corners. For now, don't worry about how well the mask conforms to the edge, just try following this one instruction: Place a point at the center of any curve and on any corner you can perceive (Figure 7.6). As using Beziers did, this takes practice, so don't worry. The mask isn't initially going to look right until you adjust tension on the points.

Figure 7.6. Your first pass with RotoBeziers should be fast and will have inaccuracies. Just place a point at the apex of each curve you find, as well as at each corner. The next step is to tighten or loosen individual points.

Okay, now how do you edit this RotoBezier? The literal key to your success is the Alt/Option key. After you've drawn your final point, keep the Pen tool and all of the mask points selected and hold down Alt/Option. You will see a double-ended arrow appear. This is the Adjust Tension pointer. Dragging it to the left increases tension (counter-intuitive, but you get used to it quickly). Try this, and all of your points become corners (Figure 7.7). Now slide it in the opposite direction, and all of the points seem to have had their Bezier handles extended to the limits.

Figure 7.7. Contrast all RotoBezier points at full tension and no tension (inset) to see how easy it is to adjust. Generally you will adjust points one or two at a time, however, not all at once.

Although it does help dispel the myth that Beziers are easier for shapes that consist mostly of corners, dragging the Adjust Tension pointer is not how you will edit RotoBeziers in most cases. Instead, after laying down the initial points of a RotoBezier mask, deselect it (Control+Shift+A/Cmd+Shift+A) and then Alt-drag the Pen tool on each point that needs its tension adjusted.

If the Selection tool (V) is active, use Ctrl+Alt (Cmd+Option) to activate the Adjust Tension pointer.

You can freely add or subtract points where needed by toggling the Pen tool (G key; look for the pen cursor to add a plus or minus sign beside it). With the Pen tool in Add mode you can also simply select a point by clicking on it, and deselect a point by Shift-clicking on it. This means that you never have to switch to another toola big advantage over Beziers (Figure 7.8).

Figure 7.8. You can use the Add Vertex tool (one of the Pen tool's modes) simply to select an existing point, without creating a new one. This means that you can always work with the Pen tool active, using the G key to cycle through its variations.

But the thing you probably still don't like about RotoBeziers is that the shape is so dissatisfying when you first lay in your points. So now that you're used to the basics of how RotoBeziers work, try drawing another mask this way: When you place a point on a corner, before moving on to draw the next point, Alt/Option-drag to the left on the corner point. Voila: It's already a corner, and your shape is much closer to final on the first pass (Figure 7.9). As you become more skilled with RotoBeziers you may even become adept at preselecting other tensions, although it's not necessary.

Figure 7.9. While drawing a RotoBezier shape, you can add a corner on the fly. Before moving on to the next point, Alt-drag (Option-drag) to the left to tighten the corner point. Draw the next point or two, and you see clearly that you have a corner and that your shape is already more accurate.

Or you can just draw the initial shape as a Bezier, if you prefer that, and consider context-clicking to change it to RotoBezier once you're done if you think you're going to want to animate that way. The real advantages of RotoBeziers show up when you animate them; you take advantage of their ability to resolve transitions with fewer variables than Beziers.

Strategies for Complex Shapes

Whether you've been won over to RotoBeziers for creating animated masks or not, the strategies for masking a complex moving shape are largely the same. Here are the main points:

Start with whatever matte you can pull to keep as much of the edges as you can. If no color key is possible, carefully examine all three channels of the footage for the one with the greatest contrast and pull a hi-con matte.

Although each new mask keeps the same color as its predecessor by default, your life will be made easier with overlapping mattes if you give each one its own color, and if you choose colors that are easy to see against your source. In the Timeline window, select the masked layer and press M to reveal its masks, then click on the colored square beside each mask to edit its color in the user interface.

Narrow down the work area to just the section you need for your shot. Do not rotoscope even one frame more than you need.

Working Quickly

Here are some tips for working more efficiently with masks:

The G key activates the Pen tool and cycles through its modes to add and subtract mask points.
Alt/Option activates the Adjust Tension slider, if the Pen tool is active.
Ctrl+Shift+A deselects the active mask; Ctrl+A activates all masks.
The V key activates the Selection tool. To switch a mask to Translate mode, double-click it with this tool; your mask will transition to Free Transfer mode and be outlined by a gray box with a center that you can offset for rotation and scaling.
With the Selection tool active, context-click on a mask for quick access to most of its editable properties.
If you have a scroll wheel on your mouse, scrolling with the Layer window active zooms in and out of frame. Holding down the spacebar activates the Hand tool for fast panning around the frame.
To replace a Mask shape, select the shape from the Layer window's Target menu and start drawing a new one; whatever you draw replaces the previous shape, creating a Mask Shape keyframe if the stopwatch is enabled. Beware: The first vertex point of the two shapes may not match, creating strange in-between frames. To set the first vertex point, context-click on the mask.

Use multiple overlapping masks instead of a single mask on a complex, moving, organic shape (Figure 7.10).

Figure 7.10. One key to working quickly is to intentionally create overlapping, separate mask shapes for different parts of your masked area that you know will move independently of one another. As a guideline, if any one shape has more than 20 or so points, you'll have difficulty retaining accuracy over time.

Go through the clip, and block in the major keyframes for a given shape. This will be the point at which a change of direction, speed, or shape begins or ends.

Start a mask on the frame on which you can draw it with the fewest points, and add points as needed to progressively more complex frames. Be prepared to break down the shot as much as necessary so that each of your masks describes only one part of the shape, and each shape stays simple. Stay under a dozen points if you can.

Block in and refine the first shape, then refine as necessary. Check the result at full speed against a contrasting background, just as you do with color keys. Then move on to the next shape.

In some cases, you can finish the matte this way. In other cases, you'll be left with a few stray holes in your matte here and there that you can go back and clean up with paint strokes. The process for working with paint is detailed in the next section; first, take a look at some of the other strategies at your disposal for masking and evaluate their effectiveness.