First you need to get Maya set up for generating normal maps.
Go to Window > Settings/Preferences > Plug-in Manager. You will see the Plug-in Manager shown in Figure A.1. This is where you control the plug-ins that are loaded into memory. Work your way down the list until you reach the plug-in called TransferSurfaceInfo.mll. Check the boxes for "loaded" and "auto load"; the plug-in will be automatically loaded the next time you boot up Maya.
Figure A.1. The Plug-in Manager
With this plug-in enabled, you can use the Transfer Surface Information tool found in the Lighting/Shading menu. We will cover this shortly.
To generate a normal map, you need to create two objects:
A higher-resolution mesh, from which you will extract the normal map information
A lower-resolution model, to which you will apply the normal map
Create the first of two polygon planes. Go to Create Polygon Primitives > Plane. Set both Subdivisions Along Width and Subdivisions Along Height to 20. This will give you the high-resolution mesh to work with (Figure A.2, top).
Figure A.2. Create the first, high-resolution polygon plane and give it some detail.
Call this new plane HighRes. Then move it across in the X axis and edit it, raising the vertices as shown in Figure A.2, bottom, to give it some detail. You just need something simple for this example; we are only using the geometry to demonstrate how normal maps work.
Create a second polygon plane, setting Subdivisions Along Width and Subdivisions Along Height to 1. This will be the low-resolution plane, to which the normal map will be applied. Call this plane LowRes.
Delete the history on both planes.
You now have your basic models (Figure A.3). Our aim is to get the same detail into the lower-resolution model that we have in the higher-resolution one.
Figure A.3. The high-resolution and low-resolution models
Before we can generate the normal map, both sets of geometry must exist in the same place. This is so that Maya knows which object the normals will be generated from as well as what UVs to fit into on the lower-resolution mesh.
Select the HighRes model, and set the Translate X attribute back to 0.
Using the Outliner, select the HighRes model. Then, holding Shift, select the LowRes model.
Switch to the Rendering menu set, and go to Lighting/Shading > Transfer Surface Information. Open the options (Figure A.4).
Figure A.4. Transfer Surface Information options
The Transfer Surface Information tool will transfer surface information from a model to an image file. We can use it to generate our normal map.
Under Target Options, set Transfer to export the Tangent Space Normals. These normals are evaluated locally and can be rotated to point in the direction of the polygon.
Object Space normals always point the same direction, so they are no good for defining our map. (Make sure to consult your lead programmer on which coordinate system to use; there are some appropriate uses for object space normals.)
The Map Width and Map Height values define the size of the bitmap image; 256x256 is fine for this example.
Specify a filename and location, and the format for the image to be exported.
Set Shading Network to "Preserve current shading network." You will need to edit the shader when you apply the normal map, so there is no point in creating new ones.
Now set the Algorithm Options:
Set Search Method to "Use outermost intersection." This controls how each object's intersections are handled.
Set Search Depth to 2. This specifies the farthest distance to which the algorithm will go to look for these intersections.
Click on Bake and Close, and the normal map will be generated.
Your resulting normal map will look like the one in Figure A.5, although it will be predominantly in blue and not grayscale.
Figure A.5. The normal map
The normal map is now exported. We can apply it to the lower-resolution model and view it interactively in Maya.