Understanding Code Optimization

Many developers take the approach of turning off the optimization switch during development and testing, and turning it on when compiling for a production release. This should in theory give you faster production code, but there is some risk with this approach. If the compiler is overzealous in its optimization or makes assumptions that are different from those made by the developers, you run the risk of seeing a bug that only appears in production code, but not during development or testing. This means you have to weigh the relative risks of an optimization bug that appears only in production versus the speed of your application. Here's a good example of this behavior:

 Option Strict On Module Module1   Sub Main()       Dim dblOne As Double = Math.Log(8, 2)       Dim dblTwo As Double = Math.IEEERemainder(dblOne, 1)       'DEBUG or RELEASE configuration #If Debug = True Then       Console.WriteLine("DEBUG configuration") #Else       Console.WriteLine("RELEASE configuration") #End If       'Started with F5 or Ctrl-F5?       Console.WriteLine("Debugger attached? " &                                   Debugger.IsAttached.ToString)       Console.WriteLine("dblOne = " & dblOne.ToString)       Console.WriteLine("dblTwo = " & dblTwo.ToString)       Console.WriteLine("dblTwo = 0? " & (dblTwo = 0).ToString)       Console.ReadLine()    End Sub End Module 

Investigating this small program is very instructional. First you should build the program in Visual Studio using the default Debug configuration, in other words without code optimization. If you then execute the program under the Visual Studio debugger, by pressing F5, you'll see a result of true printed to the console. If you try executing the program without a debugger, by pressing Ctrl-F5, you'll see a result of false instead!

Then build the program using the default Release configuration, in other words using code optimization. Once again, execute the program under the Visual Studio debugger by pressing F5. Just like before, you'll see a result of true printed to the console. If you now try executing the same program without a debugger, by pressing Ctrl-F5, you'll see a result of false .

So regardless of whether code optimization was used, running this program under the Visual Studio debugger always produces a result of true and running the same program without a debugger always produces a result of false . Is this really a code optimization issue or is it an issue related to running under the debugger? Or is it a combination of the two? It's worth noting in this context that the Visual Studio debugger always turns off code optimization, regardless of whether code optimization was specified when the program was originally compiled.

To investigate this interesting issue further, I used the console debugger Cordbg . I discuss how to use this powerful low-level debugger in Chapter 5, but one useful trick that Cordbg can do is switch code optimization on and off at will. You can't do this with the Visual Studio debugger, because it always switches off code optimization. When I ran this program under Cordbg and experimented with the code optimization settings, I got some interesting results.

The first realization was that code optimization, just like code compilation, is a two-stage process. When you build a program with code optimization, the resulting CIL is optimized. When you run a program with code optimization, the resulting native code is optimized. By default, the design-time code optimization setting for the language compiler is carried forward and used as the runtime code optimization setting for the JIT compiler (see the discussion of the DebuggableAttribute class in Chapter 3). However, Cordbg can operate to switch the JIT code optimization on and off at will, regardless of the code optimization setting used to build the program.

When I ran a default Debug build of the program under Cordbg without JIT code optimization, I saw a result of true , just like the corresponding test with the Visual Studio debugger. When I ran the same build with JIT code optimization, I saw the result of false , just as though I was running without any debugger. I got exactly the same results when I ran a default Release build of the program with and without JIT code optimization.

After the dust has settled from this experiment, it's clear that CIL code optimization (by the VB .NET compiler) doesn't affect the result produced by this program, but native code optimization (by the JIT compiler) definitely does. This is the sort of tricky issue that code optimization can produce, and you're still left with trying to understand why the native code optimization makes such a difference in this case. My guess is that it's something related to numeric precision and the use of the Double type, but this would need further investigation.

The initial results of this experiment were rather confusing, and it needed some deeper investigation to find out what was really happening. This is a good example of the sort of subtle optimization problems that can trap you. The real "gotcha" here is that this code is likely to run as expected during development and unit testing, but start producing a different result during integration testing and production.

Optimization Is Your Enemy

My advice is that before you turn on optimization automatically in your release build, you should study the performance effects of optimization on your application. Before you start sharpening your Outlook in preparation for a critical e-mail, hear me out for a minute. The odds are that allowing the compiler to optimize your code may not always result in a significant performance boost. This is because much of the code being executed is within the .NET Framework library, which is already optimized. Additionally, unless your application is performing numerically intensive work, it is quite likely that it's spending much of its time waiting for users or other resources, and there's little advantage to be gained by asking the compiler for optimization. Finally, the compiler is quite likely to optimize parts of your code that have no significant effect on the overall performance of your application. Performance bugs are notorious for being difficult to find and understand, and trying to gain a performance boost by using compiler optimizations without a prior in-depth understanding of the performance profile of your application is a recipe for nasty bugs .

If you do insist on optimizing your production build, I suggest that you then use a profiling tool in order to establish that your application is really receiving a significant performance boost in exchange for the extra risk of optimization defects. You might be surprised by what you see.

A Code Optimization Test

You can use the sorting application to test the effects of optimizing the code performance. If you run four tests for each of the four sort algorithms, you can see the performance effects of running in normal debug mode, then in debug mode but with code optimization, next in debug mode but removing integer overflow checks, and finally testing a fully optimized build with code optimization and without the integer overflow checks. You can find all of these settings on each project's Properties ’ Configuration properties ’ Optimizations page.

Table 4-1 shows the results of performing these four tests on each of the four sort algorithms, using an unsorted array containing 30,000 items, each array item containing a value between 1 and 30,000. The test hardware was a Pentium III 800 MHz laptop with 256MB of RAM, and all times shown are in seconds.

Table 4-1: Testing Code Optimization Effects on an Array with 30,000 Items

ALGORITHM	DEBUG	OPTIMIZED	NO CHECKS	OPTIMIZED/NO CHECKS
Bubble sort	10.5	9.8	9.3	5.0
Selection sort	8.0	6.1	3.7	3.1
Quick sort	1.1	1.1	1.1	0.9
Counting sort	0.0	0.0	0.0	0.0

The most interesting observation here is that the performance benefits of each of the code optimizations varies widely depending on the type of algorithm. Just turning on code optimization, as you might normally do when building in release mode, improved the bubble sort by about 7%, the selection sort by 25%, the quick sort not at all, and the counting sort was too fast to measure.

Removing the integer overflow checks improved bubble sort performance over a default debug build by 12%, but the selection sort improved by a whopping 55%. Going to full optimization finally has a performance effect on the quick sort, but this time the bubble sort benefits the most. As you can see, the benefits of code optimization vary dramatically depending on the code that's being optimized and which optimization measures are taken. Before indulging in potentially risky code optimization, you should benchmark your code to see that you're really gaining a decent benefit.

To investigate the effects of code optimization further, Table 4-2 shows the same tests, but this time done on an array containing 60,000 items, each array item having a value between 1 and 60,000.

Table 4-2: Testing Code Optimization Effects on an Array with 60,000 Items

ALGORITHM	DEBUG	OPTIMIZED	NO CHECKS	OPTIMIZED/NO CHECKS
Bubble sort	41.8	40.1	34.3	20.5
Selection sort	20.0	25.0	15.1	12.6
Quick sort	2.2	2.2	2.2	1.7
Counting sort	0.02	0.02	0.02	0.01

The shock is seeing the effect of code optimization on the selection sort ”it actually decreases performance by 25%! This isn't a freak result ”the test was run many times, and the numbers didn't vary by much. I haven't investigated the reason for this, but it's very strange that code optimization produces a performance gain of 25% when selection sorting an array of 30,000 items but a loss of 25% with an array of 60,000 items. You really should test any benefits that your application expects to see from code optimization before you start using it.