Texture Arithmetic and Combination

Some years ago, textures were used to represent materials only: bricks, stones, and wood, which is what textures were designed for. At most, some brave developers encoded lighting information into them, and by multiplying a base and light texture, created fairly convincing indoors scenes. Years went by, and some interesting new effects (namely, environment and bump mapping) were described in terms of mathematical functions on textures. These textures no longer encoded material colors only, but also other properties, such as surface normals, lighting information, and so on.

Loading textures with non-RGB information was never a problem. After all, the different APIs do not perform any kind of checking on the texture's contents. The real issue was combining them using operations other than additions and not much more. A broader range of operations, which included simple algebraic operators as well as other functionality like dot products, and so on were needed.

The first attempt was the GL_COMBINE_EXT extension, which appeared in OpenGL. This extension allowed us to specify blending operations for multitexturing, so multitexturing had an expressive potential similar to regular blending operations. We could interpolate both textures with regard to the per-vertex alpha; we could add, multiply, and so on. For example, the following code (which is rather old and thus still uses the Architecture Review Board conventions in places) combines both textures using the second texture's alpha value as the interpolator:

 // layer 0 glActiveTexture (GL_TEXTURE0); glEnable(GL_TEXTURE_2D); glBindTexture(GL_TEXTURE_2D,id0); glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE); // layer 1: modulate incoming color+texture glActiveTextureARB(GL_TEXTURE1_ARB); glEnable(GL_TEXTURE_2D); glBindTexture(GL_TEXTURE_2D,id1); glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_COMBINE_EXT); glTexEnvi(GL_TEXTURE_ENV, GL_COMBINE_RGB_EXT, GL_INTERPOLATE_EXT); glTexEnvi(GL_TEXTURE_ENV, GL_SOURCE0_RGB_EXT, GL_PREVIOUS_EXT); glTexEnvi(GL_TEXTURE_ENV, GL_OPERAND0_RGB_EXT, GL_SRC_COLOR); glTexEnvi(GL_TEXTURE_ENV, GL_SOURCE1_RGB_EXT, GL_TEXTURE); glTexEnvi(GL_TEXTURE_ENV, GL_OPERAND1_RGB_EXT, GL_SRC_COLOR); glTexEnvi(GL_TEXTURE_ENV, GL_SOURCE2_RGB_EXT, GL_PRIMARY_COLOR_EXT); glTexEnvi(GL_TEXTURE_ENV, GL_OPERAND2_RGB_EXT, GL_SRC_ALPHA); 

The syntax is rather cryptic. To understand it, you must analyze each line carefully. At the beginning, we are enabling the COMBINE_EXT extension. Then, we are requesting it to do an interpolation. SOURCE0 is then set to the previous value (thus, the color from the other texture unit) and SOURCE1 to the texture's own color. SOURCE2 is used to specify the interpolation factor, which is the alpha value of the texture.

This syntax was very non-intuitive. Thus, a new mechanism, called the register combiners, was devised by NVIDIA. The register combiners basically are a set of registers that can take the output of each texturing stage and perform arithmetic operations between them. The main advantage over the combine extension is generality and a slight improvement in ease of use.

Using register combiners usually starts by enabling/disabling them with the calls:

 glEnable(GL_REGISTER_COMBINERS_NV); glDisable(GL_REGISTER_COMBINERS_NV); 

We must then understand how many combiners we have and how they are laid out. The best way to deal with this is to have one of NVIDIA's charts at hand, so you get a global picture of who is connected to whom and how you can arrange data. In the end, the combiner operation is controlled by two calls. The first call controls the inputs to the registers, whereas the second call deals with outputs. Here is the first call:

 glCombinerInputNV(GLenum stage,    GLenum portion,    GLenum variable,    GLenum input,    GLenum mapping,    GLenum componentUsage); 

Parameters are usually specified as symbolic constants. By specifying different combinations, you can perform different arithmetic operations on texture inputs. You can see each parameter's possibilities in Table 18.2.

Table 18.2. Possible Values for glCombinerInputNV

Parameter	Values
portion =	{ GL_RGB, GL_ALPHA }
variable =	{ GL_VARIABLE_A_NV, GL_VARIABLE_B_NV, GL_VARIABLE_C_NV, GL_VARIABLE_D_NV }
input =	{ GL_ZERO, GL_PRIMARY_COLOR_NV, GL_SECONDARY_COLOR_NV, GL_CONSTANT_COLOR0_NV, GL_CONSTANT_COLOR1_NV, GL_TEXTURE0_ARB, GL_TEXTURE1_ARB, GL_FOG, GL_SPARE0_NV, GL_SPARE1_NV }
mapping =	{ GL_UNSIGNED_IDENTITY_NV, GL_UNSIGNED_INVERT_NV, GL_EXPAND_NORMAL_NV, GL_EXPAND_NEGATE_NV, GL_HALF_BIAS_NORMAL_NV, GL_HALF_BIAS_NEGATE_NV, GL_SIGNED_IDENTITY_NV, GL_SIGNED_NEGATE_NV }
componentUsage =	{ GL_RGB, GL_BLUE, GL_ALPHA }

The second call controls the outputs of the registers. Here is the syntax:

 glCombinerOutputNV(GLenum stage,    GLenum portion,    GLenum abOutput,    GLenum cdOutput,    GLenum sumOutput,    GLenum scale,    GLenum bias,    GLboolean abDotProduct,    GLboolean cdDotProduct,    GLboolean muxSum); 

Outputs have a different set of parameters and possible values. They are listed in Table 18.3.

Table 18.3. Possible Values for glCombinerOutputNV

Parameter	Value
portion =	{ GL_RGB, GL_ALPHA }
abOutput, cdOutput, sumOutput =	{ GL_DISCARD_NV,GL_PRIMARY_COLOR_NV, GL_SECONDARY_COLOR_NV,GL_TEXTURE0_ARB, GL_TEXTURE1_ARB,GL_SPARE0_NV, GL_SPARE1_NV }
scale =	{ GL_NONE, GL_SCALE_BY_TWO_NV, GL_SCALE_BY_FOUR_NV, GL_SCALE_BY_ONE_HALF_NV }
bias =	{ GL_NONE, GL_BIAS_BY_NEGATIVE_ONE_HALF_NV }

In a way, we can consider register combiners as a precursor of shading languages (specifically, of fragment shaders). Today, most developers are implementing this kind of operation, such as per-pixel lighting, Bidirectional Reflectance Distribution Function (BRDFs), or bump mapping in fragment shaders, which are far more general and standard ways of achieving these results. As such, register combiners will very likely fade out as time passes. Take a look at Chapter 21, "Procedural Techniques," and you will see the difference.

Currently, a simplified combiner version has found its way into DirectX. We can specify how each texture stage is to interact with the others, in a way reminiscent of the register combiner syntax. It all revolves around the call:

 ID3Ddevice::SetTexture( stage, textureid); 

and the call:

 ID3Ddevice:: SetTextureStageState( stage, property, value) 

The first call sets the texture stage and map to be used there. Then, a series of SetTextureStageState calls set the combiner parameters. Again, the DirectX documentation is almost mandatory to take advantage of this functionality. The best way to understand what you are doing is to draw a graph with the texture stages as boxes with lines connecting them, as if the whole system was an electrical circuit. In fact, this is not so different from the real world, because these operations are actually implemented by connecting components and registers to perform the selected arithmetic operations. Take a look at the following code example, and the graph in Figure 18.6, which depicts the relationships between the different stages:

 // Phase 0: bump mapping using dot3 (explained later) D3DXVECTOR3 m_vLight; point p(0.5,1,0.5); p.normalize(); m_vLight[0]=p.x; m_vLight[1]=p.y; m_vLight[2]=p.z; DWORD dwFactor = VectortoRGBA( &m_vLight, 10.0f ); // we store the factor so we can use it to combine d3d_device->SetRenderState( D3DRS_TEXTUREFACTOR, dwFactor ); // set texture2 on stage 0 texture2->activate(0); // operation: dot product d3d_device->SetTextureStageState(0,D3DTSS_COLOROP, D3DTOP_DOTPRODUCT3); // 1st operator the texture (a normal map) d3d_device->SetTextureStageState(0,D3DTSS_COLORARG1, D3DTA_TEXTURE); // 2nd operator: the factor that encodes the light position d3d_device->SetTextureStageState(0,D3DTSS_COLORARG2, D3DTA_TFACTOR); // store results in a temporary register d3d_device->SetTextureStageState(0,D3DTSS_RESULTARG, D3DTA_TEMP); // phase 1: enter first texture pass... the previous one was saved texture1->activate (1); // pass the 1st parameter, do not perform computations d3d_device->SetTextureStageState(1,D3DTSS_COLOROP, D3DTOP_SELECTARG1 ); // 1st parameter: the texture, a grass map d3d_device->SetTextureStageState(1,D3DTSS_COLORARG1, D3DTA_TEXTURE ); // phase 2: blend with second texture using alpha map stored in per-vertex alpha texture2->activate (2); // this blends the two operators using the per-vertex alpha d3d_device->SetTextureStageState(2,D3DTSS_COLOROP, D3DTOP_BLENDDIFFUSEALPHA ); // 1st argument: current, which means the previous stage d3d_device->SetTextureStageState(2,D3DTSS_COLORARG1, D3DTA_CURRENT ); // 2nd argument: the new texture, a rocky map d3d_device->SetTextureStageState(2,D3DTSS_COLORARG2, D3DTA_TEXTURE ); // 2nd pass: modulate using bump map info d3d_device->SetTextureStageState(3,D3DTSS_COLOROP, D3DTOP_MODULATE ); // get previous stage d3d_device->SetTextureStageState(3,D3DTSS_COLORARG1, D3DTA_CURRENT); // modulate with the temporary register d3d_device->SetTextureStageState(3,D3DTSS_COLORARG2, D3DTA_TEMP ); // final pass: modulate with per-vertex diffuse light info d3d_device->SetTextureStageState(4,D3DTSS_COLOROP, D3DTOP_MODULATE ); // 1st parameter: previous stage (combined textures + bump) d3d_device->SetTextureStageState(4,D3DTSS_COLORARG1, D3DTA_CURRENT ); // 2nd parameter: per-vertex diffuse d3d_device->SetTextureStageState(4,D3DTSS_COLORARG2, D3DTA_DIFFUSE ); // here we go! d3d_device->DrawIndexedPrimitiveUP(D3DPT_TRIANGLESTRIP, 

Figure 18.6. Graph of the preceding code: the relationships between the different texture stages.

Notice how we can specify, for each stage, which operation to perform, which inputs to choose from, and where to store the result. Typical operations are adding, modulating, or computing dot3 bump maps. Operators can be textures, results from previous stages, or temporary registers we can use to store intermediate values. A stage writes its result automatically so it can be accessed by the next one using the D3DTA_CURRENT argument. But sometimes we will need to store values for stages other than the immediate next one. A good example is the bump mapping portion in the preceding code: It is generated at phase 0, but it is used at stage 3 only. Thus, we can redirect the output of any stage to a temporary register, so we can retrieve it whenever it is actually needed.

OpenGL has followed a similar path, and today the API has incorporated most of the relevant extension tokens into the glTexEnv core call. Here is a complete example, which starts by setting the env mode to combine, so we can use the texture combiners:

 glTexEnvf(GL_TEXTURE_ENV,GL_TEXTURE_ENV_MODE,GL_COMBINE); 

And these lines set the parameters for the combiner. Notice that no extensions are used:

 glTexEnvf(GL_TEXTURE_ENV,GL_COMBINE_ALPHA,GL_SUBTRACT); glTexEnvf(GL_TEXTURE_ENV,GL_SOURCE0_ALPHA,GL_TEXTURE); glTexEnvf(GL_TEXTURE_ENV,GL_OPERAND0_ALPHA,GL_SRC_ALPHA); glTexEnvf(GL_TEXTURE_ENV,GL_SOURCE1_ALPHA,GL_PREVIOUS); glTexEnvf(GL_TEXTURE_ENV,GL_OPERAND1_ALPHA,GL_SRC_ALPHA); 

Now, depending on what we are specifying, we can access different parameters. If the value we are setting is a color value (both RGB or RGBA), Table 18.4 provides the possible values for the third parameter along with the equations that govern its behavior.

Table 18.4. Possible Values and Equations for a Third Parameter

Values	Equations
GL_REPLACE	(a0)
GL_MODULATE	(a0 * a1)
GL_ADD	(a0 + a1)
GL_ADD_SIGNED	(a0 + a1 0.5)
GL_INTERPOLATE	(a0*a2 + a1*(1-a2))
GL_SUBTRACT	(a0-a1)
GL_DOT3_RGB	4*((a0r-0.5)* (a1r-0.5) + (a0g-0.5)* (a1g-0.5) + (a0b-0.5)* (a1b-0.5))
GL_DOT3_RGBA	4*((a0r-0.5)* (a1r-0.5) + (a0g-0.5)* (a1g-0.5) + (a0b-0.5)* (a1b-0.5))

Table 18.5 lists the values for the alpha combiners.

Table 18.5. Possible Values and Equations for the Alpha Combiners

Values	Equations
GL_REPLACE	(a0)
GL_MODULATE	(a0 * a1)
GL_ADD	(a0 + a1)
GL_ADD_SIGNED	(a0 + a1 0.5)
GL_INTERPOLATE	(a0*a2 + a1*(1-a2))
GL_SUBTRACT	(a0-a1)

All of the values and equations shown in Tables 18.4 and 18.5 make for a simpler, standard interface to perform texture combinations and arithmetic.