Creating a Channel Mask: The (Practical) Details

Create a channel mask in six easy steps is the theory. Of course, most images aren't as straightforward. Let's look at some tricks you can use when creating channel masks for more challenging images. Because every image presents a unique set of challenges, use these techniques as needed.

Determining Your Intent

Your intention for an image once you've masked it determines how accurate you need to make the mask. If you're preparing a cutout figure that will be placed in a page layout on white paper, your mask will be quite forgiving and may not need to be super-accurate. If, however, you're putting your cutout subject on a brightly colored background, perhaps with different lighting conditions, then every inaccuracy will be painfully obvious. If the masked subject will be composited with other layers, experiment with blend modes to conceal edge inaccuracies. Depending on the nature of your image and the layer or layers you are compositing it with, it may not be necessary to force the contrast to opaque black and white.

In the following example of a tree, the black parts of the mask are only 70% opaque in some places. This works because the original masked sky resembles the sky color of the background layer that replaces it. Not forcing the contrast makes it possible to retain as much of the fine detail in the branches and twigs as possible.

Figure 6.6. The original images (examples A and B), the finished composite (example C), and the mask (example D). Not forcing the contrast on the mask to opaque black and white retains more of the details in the tree branches. This works in this context because one blue sky is replacing another.

Choosing Your Channel

Choosing your channel is easiest when your image has a dominant color. The most obvious example is a blue sky, which can often be easily isolated using, not surprisingly, the Blue channel.

Color channels have different contrast in different color modes. If none of your RGB channels has enough contrast, try duplicating the image and converting it to CMYK to check the channels there. Once your image is in CMYK color mode, duplicate the channel with the most contrast. To use this duplicate channel, go to your original RGB image and choose Select > Load Selection, then choose the CMYK image as the source document, and select the appropriate channel from the Channel drop-down menu.

As an alternative to duplicating a channel, you may get better results using either of two options under the Image menu, Apply Image or Calculations, to make your alpha channel (a potential selection, stored in a channel as a grayscale image). Use Apply Image to combine a duplicate channel with any other channel or with itself, using the same blend mode options available for layers.

Calculations works slightly differently, letting you blend two individual channels (or two copies of the same channel) as a new channel. Both Apply Image and Calculate potentially yield the same result. For example, using Apply Image to apply the source Blue channel with a blend mode of Overlay to a duplicate (target) of the Blue channel is the same as using Calculations to blend the (source 1) Blue channel with itself (source 2) using Overlay blend mode.

For both options Overlay blend mode usually, but not always, works best, lightening values less than 50% gray and darkening values more than 50% gray.

Figure 6.7. The Apply Image (example A) and Calculations (example B) dialog boxes.

Figure 6.8. A copy of the Blue channel (example A), and a more contrasty result using Calculations to blend the Blue channel with itself using the Overlay blend mode (example B).

You can make it easier to choose your channel mask by adding more contrast to your image with a temporary adjustment layer. Add a Curves adjustment layer and pull the curve down in the shadows and up in the highlights to increase the contrast. Now, when you view your color channels one of them should have the contrast you need. Once you have made (or calculated) your duplicate channel and refined it into a mask, you can trash the adjustment layer. Note that you can also use this approach to help out any of your Selection tools.

Figure 6.9. The original image (example A), and a duplicate of its Blue channel (example B). Adding a temporary Curves adjustment layer to increase the contrast (example C) and duplicating the Blue channel using Calculations yield a much more contrasty alpha channel (example D)a better starting point for your channel mask.

Blocking Out

In step 3 of "Creating a Channel Mask," I advised that when painting on your mask you should stay well away from the edges. In some circumstances, however, you can paint up to the edges. If an edge already has a good amount of contrast, try painting in Overlay blend mode. Painting in white in Overlay blend mode turns the gray areas white, but leaves the black areas unaffected; painting in black has the opposite effect. We saw this in "Using Quick Mask to Paint Selections" in Chapter 2, "Channels."

Forcing Contrast

Make sure that what looks like black is actually black (100%) and what looks like white is actually white (0%) by opening your Info palette and taking a tour around the edges with your Eyedropper tool to note the exact gray values. You may find it helpful to Shift-click a couple of places to take color samples of your edge areas. These samples remain fixed so that you can compare the before and after values on the Info palette.

Note that if you don't want sharp edges, then forcing your mask to opaque black and white is neither necessary nor desirable.

Because the gray values of your edges may vary, some parts of the edge may require more force than others to turn them black or white. Here are the steps you can take to force contrast.

1.	Try making a selection of as much of the edge as is similar in gray values and applying a Curve; then repeat, adjusting the Curve only as much as necessary for each selection. Those of you with strong constitutions can attempt to use Curves in Arbitrary mode to draw the shape of the curve.
2.	To draw the curve, begin by locking points onto the Curve at the darkest and lightest points of the channel by Command/Ctrl-clicking with the eyedropper. Now take the Pencil tool and draw the bottom left of the curve in to the right until it is vertically aligned beneath your first sample point (highlight). Repeat for the top right of the curve until it is vertically aligned beneath your shadow sample point. You may find it helpful to click the Smooth button a couple of times to smooth out any kinks in the curve. Figure 6.10. Using the Curve pencil to draw an arbitrary curve. The anchor points on the curve indicate the sampled areas of the imageone for the light grays and one for the dark grays.
3.	Rather than force the contrast on an alpha channel, make your duplicate channel into a layer mask and work on it there. If you are combining the masked image with the layers below it, this is a more interactive way of evaluating how the image will look in the context of the overall composition. I find it useful to do my "first draft" mask as an alpha channel (and as a "base mask," I can refer back to it), and then convert it to a layer mask to further refine it, in the context of the other layers in the composition. Alternatively, turn on the visibility of your composite RGB channel so that the alpha channel appears as a color overlay. You can paint in the mask as if it were a Quick Mask. Figure 6.11. Turn on the visibility of your RGB composite channel to view the channel mask (as a color overlay) in the context of the image.
4.	Use the Median filter, one of Photoshop's noise reduction filters (Filter > Noise > Median) at its lowest value1 pixelto pick up any stray pixels on your channel mask.

Figure 6.12. Using the Median filter to remove any stray pixels, before (example A) and after (example B).