Smart Mask Interpolation (Pro)

If you're animating Mask Shape and not getting quite the results you want, it's time for a new tool. Select your mask shape keyframes and go to Window>Smart Mask Interpolation.

Mask shapes tend not to animate smoothly. Smart Mask Interpolation creates intermediate keyframes based on the settings you choose to create smoother mask shape transformations. You can modify settings in the submenu (click on the arrow on the top right of the palette).

By default After Effects matches the first vertex. A good thing to do if you're still not getting a smooth animation is to line-up the first vertex of a path . (Thanks to Trish and Chris Meyers for this tip). Choose your first mask shape keyframe, and notice that one of the vertices is slightly bigger than the others. This is the first vertex. Move to the second keyframe and notice it's the first vertex.

Choose a vertex that more closely resembles the first mask shape's first vertex and choose Layer>Mask>Set First Vertex to designate a new first vertex.

The simplified default settings in Smart Mask Interpolation are decent, and should improve interpolation. Modifying the settings yourself can help but ultimately the best thing is to use two mask shapes that are somewhat similar.