Although hand-created and animated for the most part, masks open up all kinds of possibilities in After Effects. Masks are the principal method for defining transparency regions in a clip without regard to actual pixels because they are vector shapes. This section lays down the basics for smart use of masks.
Typical Mask Workflow
There are three tools for creating masks: two basic shape tools and the Pen. To activate the Rectangular Mask tool, use the keyboard shortcut Q; press Q again to toggle the tool to the Elliptical Mask tool. If your mask doesn't conform well to a rectangle or an ellipse, you can draw it point by point with the Pen tool (G).
If you're looking for other common primitive mask shapesa rectangle with rounded corners, a hexagon, or the likeyou may be best off drawing the shape in Adobe Illustrator (if you have it), then copying and pasting it. This, however, will not work unless the preferences in Illustrator are set properly. Under Illustrator's File Handling & Clipboard preference, choose AICB with Preserve Paths checked. This is not the default. You can also select and copy paths in Photoshop for use as masks.
Whether you draw the mask in the Composition panel or the Layer window is up to you. It is somewhere between difficult and impossible to draw or see a mask accurately in the Composition panel if, say, the layer was offset and rotated in 3D space, but if you want to see the layer over its background, the Composition panel must at least be visible. Given sufficient monitor space, an ideal compromise is to keep both windows open side by side, working in the Layer window and watching the Composition panel for live updates.
Remember that your target shape doesn't have to be an ellipse to benefit from starting by drawing an ellipse. Indeed, if the shape calls for a perfectly circular curve on one side, you might do well to draw an ellipse (holding down Shift to make it perfectly circular) and then editing other sides of it with the Pen tool.
When drawing a rectangle or ellipse
Figure 3.13. Flipping a custom mask symmetrically on one axis is no easy trick in After Effects. Holding down the Shift key scales both axes proportionally, flipping it on both axes. Instead, with View > Show Grid enabled and View Snap to Grid turned on, double-click the Rectangular Mask tool to create a second mask that is the size of the layer. Select both masks in the Timeline, and double-click a point on one of the masks to set the Free Transform tool. Now drag the handles at either side of the image and swap their positions, deleting the layer-sized mask when you're done.
Highlighting a layer with a mask and pressing MM (the M key twice in rapid succession) reveals the full Mask options for that layer. Some tips for those include
Figure 3.14. The Feather of this mask is set roughly equal to its radius, creating a big, diffuse gradient in the shape of the mask (in this case, elliptical), useful for many types of lighting effects.
Keyboard shortcuts help eliminate a lot of the fuss and bother that comes with masking in After Effects.
Drawing Bezier Masks
By default, the Pen tool draws Bezier curves, and in After Effects you can do everything you need to create a mask by keeping the Pen tool active throughout the mask edit.
I sometimes draw a Bezier mask first as straight lines only, clicking to place points at key transitions and corners. Once I've completed the basic mask shape, with the Pen tool still active, I can go back point by point and edit the shape, because I have instant access to all of the mask shortcuts:
Only when you want to double-click to free transform the entire mask is it necessary to switch to the Selection tool, which you can do either by pressing V or by holding down Ctrl/Cmd while you double-click a vertex. The Ctrl/Cmd key toggles back and forth between the Pen and Selection tools, regardless of which one is currently active.
As with most selectable UI elements in After Effects, pressing F2 or Ctrl+Shift+A (Cmd+Shift+A) deselects the active mask. This is handy when you're done drawing one mask and want to create the next one without switching tools.
Combining Multiple Masks