Step-by-Step Shader Tutorial

This section goes through the whole process of creating your own shader. It will not be the most useful shader in the world, but it shows how to get shaders done, which tools can be used, and prepares you for future shaders, which might be more exciting.

To see where you are heading take a look at Figure 6-4, which shows the apple model from the last chapter, but this time your new shader SimpleShader.fx is applied onto it and another texture is used with some text on it.

image from book
Figure 6-4

The SimpleShader.fx uses a normal vertex shader, which just transforms the 3D data and a simple specular per-pixel technique for the pixel shader, which processes every pixel and calculates the ambient, diffuse, and specular components for it. This kind of calculation is not possible with a fixed function shader because you cannot program any special formulas for the pixel shader and the only way to show specular light effects is to do them in the vertex shader. You might ask why it is that bad to calculate the specu lar color component in the vertex shader. Well, if you have a low-polygon object like the apple here or the sphere from the normal mapping example earlier, just calculating color components in the vertex shader will look very bad because you can only calculate the resulting colors for every vertex. If you take a look at the wireframe for the apple (see Figure 6.5) you can see that there are only a bunch of vertices, but you have many more pixels to fill in between. If you only calculate the vertices (the points that connect the wires), all data in between is not calculated the right way and can only be interpolated. But if such a highlight as in Figure 6-4 is between two vertices, it will not be visible at all. By the way, if you want to render in wireframe mode just add the following line before you render any 3D data:

  BaseGame.Device.RenderState.FillMode = FillMode.WireFrame; 

image from book
Figure 6-5

FX Composer

To get started with the little SimpleShader.fx shader you are going to use the freely available FX Composer tool, which you can download at the Nvidia home page ( in the developers section (or just Google for it).

After you have installed and started FX Composer you will see the screen shown in Figure 6-6. It shows you several panels and can even be used by artists to test out textures, shader techniques, and to modify shader parameters like color values, effect strengths, and more. For you as the developer the most important panel is in the center, which shows the source code of the currently selected .fx file (similar to Visual Studio). As you can see the .fx file looks very similar to C# or CPP and it has syntax highlighting and many keywords, which look similar to C#. For example, string is just a text string; float is just a floating-point number, and so on, just like in C#. But there are also other types like float3 or float4x4. These numbers behind the float name just indicate the dimensions; float3 is the same structure as a Vector3 in XNA, which contains three floats too (x, y, and z). Float4x4 describes a matrix and contains 16 float values (4×4); it is also in the same format as the XNA Matrix structure. The last variable types you have to know are for textures. Textures are defined with the texture type and you also have to specify a sampler to tell the shader how to use this shader (which filtering to use, which dimensions the texture has, and so on).

image from book
Figure 6-6

Last but not least you see the optional semantics values (WorldViewProjection, Diffuse, Direction, and so on) behind variables indicated by a column. They tell FX Composer how to fill in this value and how to use it. For your XNA program these values do not matter, but it is very common to always specify these semantics. It allows you to use the shader in other programs like FX Composer, 3D Studio Max, and it is also helpful when you read through the shader later. The semantics tell you exactly which variable was intended for what use. In XNA you are not limited to the use; you can set the world matrix to the viewInverse, for example, but that would be very confusing after a while, wouldn’t it?

The other panels are not so important to you right now, but here’s an explanation of what each of them does:

  • Use the toolbar on the top to quickly load a shader and save your current file. The last button in the toolbar builds your shader and shows you any compiler errors in the log panel below (very similar to Visual Studio). Every time you build a shader it gets saved automatically and you should use this button (or the Ctrl+F7 hotkey) as often as possible to make sure your shader always compiles and is saved.

  • The Materials panel on the left side shows you a list of all shaders you have currently loaded into FX Composer with a little preview sphere, which changes as soon as you change any shader. The most important button here is the “Assign to Selection” button to set this material to the currently selected Scene panel object.

  • Textures show the textures used in the current shader. If you load external shaders, texture files are often not found because they reside in another directory or were not shipped with the .fx shader file. Textures that could not be loaded are shown as all blue bitmaps and you should make sure to load a texture first, otherwise the shader output is usually black and useless.

  • The Properties panel on the right side shows all the parameters in the shader you can set (or not, because they are already filled in by FX Composer, like world, viewInverse, and so on). To make the shader work the same way in your XNA engine you have to set all these values in your game engine too, especially the matrices, which are dependent on the current camera position, viewport, and object in the 3D world. If you don’t set a parameter the default value is used, which can be set directly in the source code of the .fx file. To make sure all parameters always have valid values, even if the user or engine does not set any of the color parameters, for example, you should always make sure the parameters have useful default settings like setting the diffuse color to white:

      float4 diffuseColor : Diffuse = {1.0f, 1.0f, 1.0f, 1.0f}; 

  • And finally, the Scene panel shows a simple example object to test your shader like the standard sphere. You can also change it to a cube, cylinder, or something else. Alternatively you can even import model files and play around with them, but most files do not work very well and the camera in FX Composer always gets messed up when you import scenes. I would just stick with the standard sphere object and do all advanced tests in your XNA engine. FX Composer 2.0 will be much better for loading and handling custom 3D data and model files. You can use Collada files and even manage all the shaders that are used by every mesh in your model. Use FX Composer 2.0 if it is available by the time you read this.

FX File Layout

If you have worked with OpenGL before and had to write your vertex and fragment shaders (which just means vertex and pixel shaders in DirectX) yourself, you will be happy to hear that .fx files have all the shader code in one place and you can have many different pixel shader blocks in your .fx files to support multiple target configurations like pixel shader 1.1 for the GeForce 3 and pixel shader 2.0 support for the Nvidia GeForce FX or the ATI Radeon 9x series. Another useful side effect of having the ability to support multiple vertex and pixel shaders in one .fx file is to put similar shaders together and have them all use some common methods and the same shader parameters, which makes shader development much easier. For example, if you have a normal mapping shader for shiny metal surfaces, which looks really good on metal, you maybe want another more diffuse looking shader for stones, and you can put another shader technique in the same shader file and then select the used technique in your engine based on the material you want to display.

Figure 6-7 shows the SimpleShader.fx file layout as an example of a typical .fx file. More complex shaders will have more vertex and pixel shaders and more techniques. You don’t have to create new vertex shaders or pixel shaders for each technique; you can combine them in any way you want to. Some shaders might also use multiple passes, which means they render everything with the first pass, and then the second pass is rendered with the exact same data again to add more effects or layers to a material. Using multiple passes is usually too slow for real-time applications because rendering with 10 passes means your rendering time will be 10 times higher than just rendering one pass with the same shader. In some cases it can be useful to use multiple passes to achieve effects that would otherwise not fit into the shader instruction limit or for post screen shaders where you can use the results from the first pass and modify them in the second pass to achieve much better effects. For example, blurring takes a lot of instructions because to get a good blurred result you have to mix many pixels to calculate every blurred point.

image from book
Figure 6-7

As an example, blurring with the range of 10×10 pixels would take 100 pixel read instructions and that does not sound good if you have one or two million pixels you want to blur the whole screen. It is much better in this case just to blur in the x direction (10 pixel read instructions) and then take the result and blur it in the y direction in the second pass with another set of 10 pixel read instructions. Now your shader runs five times faster and looks pretty much the same. You can even achieve much better performance results by first sampling down the background image from 1600×1200 to 400×300 and then perform the blurring, which gives you another performance improvement by the factor of 16 (now that is amazing).

Chapter 8 talks about post screen shaders; but first, write the SimpleShader.fx file. As you see in Figure 6-7 the shader file uses quite a lot of shader parameters. Some of them are not so important because you could also hard-code the material settings directly into the shader, but this way you have the ability to change the color and appearance of the material in your engine and you can use the shader for many different materials. Other parameters like the matrices and the texture are very important and you cannot use the shader if these parameters are not set by the engine. Material data like the color values and the texture should be loaded at the time you create the shader in your engine and the world matrices, light direction, and so on should be set each frame because this data can change every frame.


If you want to follow the creation steps of this shader closely you might want to open up FX Composer now and start with an empty .fx file. Select File image from book New to create a new empty .fx file and delete all content. You will start with a completely empty file.

The first thing you might want to do to quickly remember what this file is about when you open it up later is to add a description or comment at the top of the file:

  // Chapter 6: Writing a simple shader for XNA 

As you saw in the SimpleShader.fx overview you need several matrices first to be able to transform the 3D data in the vertex shader. This would be the worldViewProj matrix, the world matrix, and the viewInverse matrix. The worldViewProj matrix combines the world matrix, which puts the object you want to render at the correct position in the world; the view matrix, which transforms the 3D data to your camera view space (see Figure 5-7 from the previous chapter); and finally, the projection matrix, which puts the view space point to the correct screen position. This matrix allows you to quickly transform the input position with just one matrix multiply operation to get the final output position. The world matrix is then used to perform operations in the 3D world like calculating the world normal, lighting calculations, and more. The viewInverse is usually just used to get more information about the camera position, which can be extracted from this matrix by getting the 4th row:

  float4x4 worldViewProj : WorldViewProjection; float4x4 world : World; float4x4 viewInverse : ViewInverse; 

Each of these matrix values is a float 4×4 type (which is the same data type as Matrix in XNA) and you use the shader semantics to describe these values better and to have support in applications like FX Composer or 3D Studio Max, which is important when the modeler wants to see how his 3D model looks with your shader. The cool thing is that it will look absolutely the same no matter whether it is displayed in FX Composer, 3D Studio, or in your engine. This fact can save you a lot of game development time; it especially shortens the testing process to get the appearance of all 3D objects right.

Time to save your file now. Just press the Build button or Ctrl+F7 and you are prompted to enter a name for the new shader. Just name it SimpleShader.fx and put it into your XnaGraphicEngine content directory so you can quickly use it in XNA later. After saving FX Composer will tell you that “There were no techniques” and “Compilation failed” in the Tasks panel below the source code. That’s ok; you will implement the techniques soon, but first implement the rest of the parameters. Because your shader uses a light to lighten up your apple (see Figure 6-4) you need a light, which can be either a point light or a directional light. Using point lights is a little bit more complicated because you have to calculate the light direction for every single vertex (or even for every single pixel if you like). The calculation becomes even more complex if you use a spot light. Another problem with point lights is that they usually become weaker over the distance and you will need a lot of lights if your 3D world is big. Directional lights are much easier and are very useful to quickly simulate the sun in an outside environment, like for the game you will create in the next chapters.

  float3 lightDir : Direction <   string Object = "DirectionalLight";   string Space = "World"; > = { 1, 0, 0 }; 

In addition to ambient lighting, which just adds a general brightness to all materials, the following light types are commonly used in games:

  • Directional Lights: Simplest light type, very easy to implement. You can directly use the light direction for the internal lighting calculations in shaders. In the real world no directional lighting exists; even the sun is just a big distant point light, but it is much easier to implement it for outdoor lighting scenes.

  • Point Lights: Calculating a single point light is not much harder, but you have to calculate the falloff to make the light weaker over distance, and in case you need the light direction it needs to be calculated in the shader too, which slows things down. But the main problem for point lights is that you need more than one light for any scene bigger than just a room. 3D shooters usually use tricks to limit the amount of point lights that can be seen at the same time, but for outdoor games like strategy games it is much easier to use a directional light and just adding maybe a few light effects and simple point lights for special effects.

  • Spot Lights: These are the same thing as point lights, but they point in just one direction and only lighten up a small spot with help of a light cone calculation. Spot lights are a little bit more difficult to calculate, but if you are able to skip the hard part of the lighting calculation (for example, when using a complex normal mapping shader with multiple spot lights), it can even be a lot faster than using point lights. Currently you can only do conditional statements like “if” in Shader Model 3.0, previous Shader versions support these statements too, but all “if” statements and “for” loops will just be unrolled and expanded, you will not gain a great performance benefit like on Shader Model 3.0.

Now the preceding code does look a little bit more complicated; the first line is pretty much the same as for the matrices. float3 specifies that you use a Vector3 and Direction tells you that lightDir is used as a directional light. Then inside the brackets an Object and a Space variable is defined. These variables are called annotations and they specify the use of the parameter for FX Composer or other external programs like 3D Studio Max. These programs now know how to use this value and they will automatically assign it to the light object that may already exist in the scene. This way you can just load the shader file in a 3D program and it will work immediately without having to link up all light, material settings, or textures by hand.

Next you are going to define the material settings; you are going to use the same material settings that a standard DirectX material uses. This way you can use similar shaders or older DirectX materials in programs like 3D Studio Max and all color values are automatically used correctly. In the engine you usually will just set the ambient and diffuse colors and sometimes you also specify another shininess value for the specular color calculation. You might notice that you did not use any annotations here - you can specify them too here, but material settings work fine in both FX Composer and 3D Studio Max even if you don’t define annotations. The engine can also just use the default values in case you don’t want to overwrite the default values for your unit test later.

  float4 ambientColor : Ambient = { 0.2f, 0.2f, 0.2f, 1.0f }; float4 diffuseColor : Diffuse = { 0.5f, 0.5f, 0.5f, 1.0f }; float4 specularColor : Specular = { 1.0, 1.0, 1.0f, 1.0f }; float shininess : SpecularPower = 24.0f; 

Last but not least your shader needs a texture to look a little bit more interesting than just showing a boring gray sphere or apple. Instead of using the apple texture from the original apple like in the last chapter, you are going to use a new test texture, which will become more interesting in the next chapter when you add normal mapping. The texture is called (see Figure 6-8):

  texture diffuseTexture : Diffuse <   string ResourceName = ""; >; sampler DiffuseTextureSampler = sampler_state {   Texture = <diffuseTexture>;   MinFilter=linear;   MagFilter=linear;   MipFilter=linear; }; 

image from book
Figure 6-8

The ResourceName annotation is only used in FX Composer and it automatically loads the file from the same directory the shader is in (make sure the file is in the XnaGraphicEngine content directory too). The sampler just specifies that you want to use linear filtering for the texture.

Vertex Input Format

Before you can finally write your vertex and pixel shader you have to specify the way vertex data is passed between your game and the vertex shader, which is done with the VertexInput structure. It uses the same data as the XNA structure VertexPositionNormalTexture, which is used for the apple model. The position is transformed with the worldViewProj matrix defined earlier in the vertex shader later. The texture coordinate is just used to get the texture coordinate for every pixel you are going to render in the pixel shader later and the normal value is required for the lighting computation.

You should always make sure that your game code and your shaders are using the exact same vertex input format. If you don’t do that the wrong data might be used for the texture coordinates or vertex data might be missing and mess up the rendering. The best practice is to define your own vertex structure in both the application (see TangentVertex in the next chapter) and then define that same vertex structure in the shader. Also set the used vertex declaration, which describes the vertex structure layout, in your game code before calling the shader. You can find more details about that in the next chapter.

  struct VertexInput {   float3 pos : POSITION;   float2 texCoord : TEXCOORD0;   float3 normal : NORMAL; }; 

In a similar way you have also to define the data that is passed from the vertex shader to the pixel shader. This may sound unfamiliar at first and I promise you this is the last thing you have to do to finally get to the shader code. If you take a look at Figure 6-9 you can see the way 3D geometry is traveling from your application content data to the shader, which uses the graphic hardware to finally end up on the screen in the way you want it to be. Though this whole process is more complicated than just using a fixed function pipeline like in the old DirectX days, it allows you to optimize code at every point, and you can modify data in every single step (either at the application level, or dynamically when the vertices are processed in the vertex shader or you can modify the color of the final pixel when it is rendered on the screen).

image from book
Figure 6-9

The VertexOutput structure of your shader passes the transformed vertex position, the texture coordinate for the used texture, and a normal and halfVec vector to perform the specular color calculation directly in the pixel shader. Both vectors have to be passed as texture coordinates because passing data from the vertex to the pixel shader can only be position, color, or texture coordinate data. But that does not matter; you can still use the data the same way as in the VertexInput structure. It is very important to use the correct semantics (Position, TexCoord0, and Normal) in the VertexInput structure to tell FX Composer, your application, or any other program that uses the shader.

Because you define the VertexOutput structure yourself and it is only used inside the shader, you can put anything you want in here, but you should keep it as short as possible and you are also limited by the number of texture coordinates you can pass over to the pixel shader (four in pixel shader 1.1, eight in pixel shader 2.0).

  struct VertexOutput {   float4 pos : POSITION;   float2 texCoord : TEXCOORD0;   float3 normal : TEXCOORD1;   float3 halfVec : TEXCOORD2; }; 

Vertex Shader

The vertex shader now takes the VertexInput data and transforms it to the screen position for the pixel shader, which finally renders the output pixel for every visible polygon point. The first lines of the vertex shader usually look very much the same as for every other vertex shader, but you often pre-calculate values at the end of the vertex shader for use in the pixel shader. If you are using pixel shader 1.1 you cannot do certain things like normalizing vectors or execute complex mathematical functions like power. But even if you use pixel shader 2.0 (like you are for this shader) you might want to pre-calculate certain values to speed up the pixel shader, which is executed for every single visible pixel. Usually you will have far fewer vertices than pixels, and every complex calculation you make in the vertex shader can speed up the performance of the pixel shader several times.

  // Vertex shader VertexOutput VS_SpecularPerPixel(VertexInput In) {   VertexOutput Out = (VertexOutput)0;   float4 pos = float4(In.pos, 1);   Out.pos = mul(pos, worldViewProj);   Out.texCoord = In.texCoord;   Out.normal = mul(In.normal, world);   // Eye pos   float3 eyePos = viewInverse[3];   // World pos   float3 worldPos = mul(pos, world);   // Eye vector   float3 eyeVector = normalize(eyePos-worldPos);   // Half vector   Out.halfVec = normalize(eyeVector+lightDir);   return Out; } // VS_SpecularPerPixel(In) 

The vertex shader takes the VertexInput structure as a parameter, which is automatically filled in and passed from the 3D application data by the shader technique you will define later at the end of the .fx file. The important part here is the VertexOutput structure, which is returned from the vertex shader and then passed to the pixel shader. The data is not just passed 1:1 to the pixel shader, but all values are interpolated between every single polygon point (see Figure 6-10).

image from book
Figure 6-10

This is of course a good thing for any position and color values because the output looks much better when values are interpolated correctly. But in case you use normalized vectors they can get messed up in the process of the interpolation that the GPU does automatically. To fix this you have to re-normalize vectors in the pixel shader (see Figure 6-11). Sometimes this can be ignored because the artifacts are not visible, but for your specular per pixel calculation it will be visible on every object with a low polygon count. If you are using pixel shader 1.1 you can’t use the normalize method in the pixel shader. Instead you can use a helper cube map, which contains pre-calculated normalized values for every possible input value. For more details please take a look at the NormalMapping and ParallaxMapping shader effects of the next chapters.

image from book
Figure 6-11

If you take a quick look at the source code again (or if you are typing it in yourself to write your first shader), you can see that you start by calculating the screen output position. Because all matrix operations expect a Vector4 you have to convert your input Vector3 value to Vector4 and set the w component to 1 to get the default behavior of the translation part of the worldViewProj matrix (translation means movement for matrices).

Next the texture coordinate is just passed over to the pixel shader; you are not interested in manipulating it. You can, for example, multiply the texture coordinates here or add an offset. Sometimes it can even be useful to duplicate the texture coordinate and then use it several times in the pixel shader with different multiplication factors and offsets for detail mapping or water shaders.

Each normal vector from the apple model is then transformed to the world space, which is important when you rotate the apple model around. All normals are rotated then as well, otherwise the lighting will not look correct because the lightDir value does not know how each model is rotated, and the lightDir values are just stored in the world space. Before applying the world matrix your vertex data is still in the so-called object space, which can be used for several effects too, if you like to do that (for example, wobble the object around or scale it in one of the object directions).

The last thing you do in the vertex shader is to calculate the half vector between the light direction and the eye vector, which helps you with calculating the specular color in the pixel shader. As I said before it is much more effective to calculate this here in the vertex shader instead of re-calculating this value for every single point over and over again. The half vector is used for phong shading and it generates specular highlights when looking at objects from a direction close to the light direction (see Figure 6-12).

image from book
Figure 6-12

Pixel Shader

The pixel shader is responsible for finally outputting some pixels on the screen. To test this you can just output any color you like; for example, the following code outputs just a red color for every pixel that is rendered on the screen:

  // Pixel shader float4 PS_SpecularPerPixel(VertexOutput In) : COLOR {   return float4(1, 0, 0, 1); } // PS_SpecularPerPixel(In) 

If you press Build now the shader still does not compile because you have not defined a technique yet. Just define the following technique to get the shader to work. The syntax of techniques is always similar; usually you just need one pass (called P0 here) and then you define the used vertex and pixel shaders by specifying the used vertex and pixel shader versions:

  technique SpecularPerPixel {   pass P0   {     VertexShader = compile vs_2_0 VS_SpecularPerPixel();     PixelShader = compile ps_2_0 PS_SpecularPerPixel();   } // pass P0 } // SpecularPerPixel 

Now you are finally able to compile the shader in FX Composer and you should see the output shown in Figure 6-13. Make sure you have the shader assigned to the selection in the Scene panel in FX Composer (click the sphere, then click the SimpleShader.fx material in the Materials panel and click the “Apply Material” button).

image from book
Figure 6-13

The next step you should take is to put the texture onto the sphere. This is done with the help of the tex2D method in the pixel shader, which expects a texture sampler as the first parameter and the texture coordinates as the second parameter. Replace the return float4 line from the previous code with the following code to texture your 3D object:

  float4 textureColor = tex2D(DiffuseTextureSampler, In.texCoord); return textureColor; 

After compiling the shader you should now see the result shown in Figure 6-14. If you just see a black sphere or no sphere at all you probably don’t have the texture loaded (see the Textures panel and make sure the texture is loaded as described previously; you can click the diffuseTexture in properties and load it yourself).

image from book
Figure 6-14

The last thing you have to do is to calculate the diffuse and specular color components based on the lightDir and halfVec values. As mentioned, you also want to make sure the normals are re-normalized in the pixel shader to get rid of any artifacts.

  // Pixel shader float4 PS_SpecularPerPixel(VertexOutput In) : COLOR {   float4 textureColor = tex2D(DiffuseTextureSampler, In.texCoord);   float3 normal = normalize(In.normal);   float brightness = dot(normal, lightDir);   float specular = pow(dot(normal, In.halfVec), shininess);   return textureColor *     (ambientColor +     brightness * diffuseColor) +     specular * specularColor; } // PS_SpecularPerPixel(In) 

The diffuse color is calculated by taking the dot product of the re-normalized normal (in world space, see the vertex shader discussion earlier in this chapter) and the lightDir, which is also in world space. It is always important that you are in the same space if you do any matrix, dot, or cross product calculations, otherwise the results will be very wrong. The dot product behaves just the way you need for the diffuse color calculation. If the lightDir and the normal point in the same direction it means that the normal is pointing right at the sun and the diffuse color should be at the maximum value (1.0); if the normal is at a 90 degree angle, the dot product will return 0 and the diffuse component is zero. To even see the sphere from the dark side the ambient color is added, which lights up the sphere even when no diffuse or specular light is visible.

Then the specular color is calculated with the Phong formula to take the dot product of the normal and the half vector you calculated in the vertex shader. Then you take the power of the result by the shininess factor to make the area much smaller that is affected by the specular highlight. The higher the shininess value, the smaller the highlight gets (play around with the shininess value if you want to see it). At the end of the pixel shader you can finally add all color values together and you multiply the result by the texture color and return everything to be painted on the screen (see Figure 6-15).

image from book
Figure 6-15

You are now finished with working on the shader. The saved shader file can now even be used in other programs like 3D Studio Max to help artists to see how the 3D models will look in the game engine. Next you are going to implement the shader into your XNA engine.

Professional XNA Game Programming
Professional XNA Programming: Building Games for Xbox 360 and Windows with XNA Game Studio 2.0
ISBN: 0470261285
EAN: 2147483647
Year: 2007
Pages: 138

Similar book on Amazon
XNA Game Studio 4.0 Programming: Developing for Windows Phone 7 and Xbox 360 (Developer's Library)
XNA Game Studio 4.0 Programming: Developing for Windows Phone 7 and Xbox 360 (Developer's Library)
Learning XNA 3.0: XNA 3.0 Game Development for the PC, Xbox 360, and Zune
Learning XNA 3.0: XNA 3.0 Game Development for the PC, Xbox 360, and Zune
Beginning XNA 2.0 Game Programming: From Novice to Professional (Expert's Voice in Game Programming)
Beginning XNA 2.0 Game Programming: From Novice to Professional (Expert's Voice in Game Programming)
Microsoftu00ae XNAu00ae Game Studio 3.0: Learn Programming Now! (Pro - Developer)
Microsoftu00ae XNAu00ae Game Studio 3.0: Learn Programming Now! (Pro - Developer) © 2008-2017.
If you may any questions please contact us: