New to After Effects 6 is the Auto-trace option. You can convert the alpha, luminance, red, green, or blue channels of a layer to one or more masks. Once you have a mask, you can apply any mask-based effects.
Auto Trace is work-area sensitive so adjust your timeline accordingly . Instead of dragging the work area tabs, hit B to begin the workspace, and N to end it.
Select your layer and choose Layer>Auto-trace to bring up the options box.
For a less detailed trace, increase your pixel tolerance; for more detail, decrease tolerance. More detail means more time.
Threshold is used to determine where to draw the mask, and is really intended only for layers without an alpha. Values over the percentage you enter are mapped to white, and made opaque . Values under the threshold are mapped to black, and are made transparent.
You may find it useful to blur the image a few pixels before tracing. This will generate a mask with fewer points.