Not to be confused with blend modes, Blending Options appears at the top of the Layer Style dialog and controls the way that layers interact with each other. To illustrate this, I have taken a photo and added a layer that has squares filled with black, white, 50% gray, and gradients.
Double-clicking beside the layer name will open the Layer Style dialog with the focus on Blending Options: Default. (You can also Control-click [PC: Right-click] on the layer and choose Blending Options at the top of the contextual menu.) The two sliders under Blend If are the controls we'll use to change the interaction of the layers.
Using the top slider called This Layer means (strangely enough) that you will be affecting the current layer. Here I dragged the black slider slightly to the right. That made anything black on the layer become see-through, showing the layer below. Notice how the black portion of the gradient boxes also became see-through.
In this case, I took the white slider under This Layer and pulled it in to the left, making the white areas of the layer see-through. Since the boxes were pure white, they disappeared as soon as I moved the slider even a small amount. Again, take note of the white portions of the gradient boxes and how those areas were affected. If the layer contains white and shades that are progressively darker than white, the further I drag the white slider, the more shades will be affected.
One of the potential problems with these sliders is the tendency to get very obvious edges where the colors become see-through. To fix this, press-and-hold the Option key (PC: Alt key) and click on the slider to split it into two. Now, as the two halves of the slider are moved, it will create a smoother transition on the edges.
The same concept applies to the Underlying Layer section: as you drag the sliders, the dark or light areas of the bottom layer start to show through the top layer. Here I dragged the black slider to the right to force dark areas of the Background layer to show through the top layer.
In the next case, the white slider was moved way to the left to make the few lighter areas of the shed show through the squares.
In an example of what will often be your strategy, I then split the two sliders and played with the sides of both the white and black sliders until I got the result I wanted.
Another very effective way to use the Blend If sliders is with an adjustment layer. Here I applied a Color Balance adjustment layer to change the look of the photo. Then I Control-clicked (PC: Right-clicked) on the adjustment layer, chose Blending Options, and used the methods previously described to change the areas of the photo that were affected by the adjustment layer.