This chapter talked about a lot of post-screen shaders and covered almost every basic post-screen shader that exists, and you learned about the basic components to write similar shaders. For example, the color correction shader can be used to achieve sepia effects like in Figure 8-1. One shader I left out intentionally is the edge detection shader. It may not be useful for every single game, but there are a lot of other shaders that use the basic functionality of an edge detection shader. Here are some examples:
Edge detection shaders for an evil alien-like look or for a high-tech special ops view mode that allows you to see much sharper in the night.
Comic and toon shaders, which just take the input color and reduce it to very few colors, for example from a color table or simple image that maps each color value to certain predefined colors. Then the thick border lines are drawn for every 3D surface, which can be done via an edge detection shader quite easily. You could also improve it a bit by drawing the lines in 3D and then adding the solid geometry on top of it, leaving you with the outlines for each 3D object, which the edge detection shader can pick up more easily.
Edge detection shaders can be used to add contrast to the scene or to find lines or contours. Edge detection is often the opposite of a blur effect, which mixes everything together making the final result sometimes washed out too much. An edge detection shader can help if you want a sharper image to start with before doing additional glow and blur effects.
Edge detection shaders can also be used to create very artistic images that look like they were painted and not rendered. You just use a very high edge detection amount and add other filters like strain effects or the half-tone shader from Figure 8-1.
All you have to do is to create a simple edge detection filter. The basic idea is here to take several pixels from the original scene map and compare them with the neighboring pixels. In Shader Model 1.1 you can only take four texture inputs, and in Shader Model 2.0 only eight, therefore you are limited on how many pixels you can use in the Pixel Shader. Doing comparisons is also not that easy and not a good thing for Pixel Shader 1.1. Instead just take a center point, multiply it by 2, and then subtract two surrounding points (top and bottom, and left and right). Then take the gray-scale value with the help of the Luminance function of the result and you have your edges.
The following is the basic Pixel Shader code; if you are in Shader Model 2.0 you can do even more texture fetches. If you are unsure how to extend the shader please check out the edge detection shader of Nvidia’s FX Composer. You should try to develop the whole shader in FX Composer anyway and then you can work on a C# class if you want to import it into your engine. The variables col1, col2, col3, and col4 are four colors from the scene map through four texture coordinates you passed to the Pixel Shader.
return Luminance( // Horizontal (col1*2.0f-col2-col3)+ // Vertical (col2*2.0f-col1-col4));
If you managed to get your shader to work you should see something similar to Figure 8-24.