Three Debugging Modes

The VB.Classic environment allowed only one way to debug a program. You loaded the source code into the IDE and the VB interpreter executed the p-code (packed code, an intermediate format between source and native code). This was a simple concept and therefore easy to understand, even for a VB neophyte. From version 5.0 onward, you could also compile your source code into processor-native code, but then you were on your own if you wanted to debug the result.

The .NET environment gives you no less than three ways of debugging your application, and each of these ways has some variations. As happens so often in the .NET world, you now have more choices, more flexibility, and more complexity.

• IDE debugging : You start an application within Visual Studio, and the Visual Studio debugger automatically attaches to the processes within that application.

• PROCESS debugging : You manually attach a debugger to a process that's already running.

• JIT debugging : The CLR prompts you to attach a debugger to an application that has crashed.

The first method is similar to the VB.Classic way of doing things, although it has its own quirks . The other two methods will be unfamiliar to many VB.Classic developers, but they're useful because of the extra debugging facilities that they make possible. In fact, all three methods work the same way under the hood in that they all involve a debugger being attached to the process that you want to debug. Visual Studio does the process attachment automatically with the first method, you do the process attachment manually with the second method, and the final method requires you to tell the CLR which debugger you want to attach to a crashed process. The next three sections discuss how each of these three methods works when using the Visual Studio debugger.

IDE Debugging

This is relatively familiar territory to a VB.Classic developer. You load the solution that contains your application into Visual Studio, set your breakpoints, and press F5. Visual Studio automatically attaches its debugger to your application's processes, and as long as your debugging doesn't step outside of the projects that make up your Visual Studio solution, you can step through your code and debug in comfort .

A twist on this process is the ability to start more than one of the projects in your solution simultaneously . If you're executing components that need to run side by side and cooperate with each other, this is now very easy. In the Solution Explorer window, right-click your solution, select Set Startup Projects ’ Common Properties ’ Startup Project, and click the Multiple Startup Projects option. This allows you to specify which of the projects that make up your solution need to start together. If you examine the Solution Explorer window, a solution with a single startup project shows that startup project in bold. If you use the Multiple Startup Projects option, the solution itself is shown in bold instead.

An important point is that you can combine the debugging of your current Visual Studio solution with the "attach to process" method of debugging explained in the next section. Assuming that a project within your Visual Studio solution calls into another process to which you've attached the debugger, you can step seamlessly back and forth between your code in your solution and the code in the process that you've attached to manually.

You can also use the IDE to debug an application that was built outside of Visual Studio or one for which you have no source code. If you don't have a solution open , or you don't want to add the application to an existing solution, select File ’ Open ’ File and find and select the .exe file. To add the application to an existing solution, choose File ’ Add Project ’ Existing Project and then select the .exe file. Once the .exe file is loaded and shown in the Solution Explorer window, right-click the imported.exe file and select Debug ’ Step into new instance, and Visual Studio attempts to find any associated source and debug symbol files so that you can start debugging. Alternatively, you can use the menu option File ’ Open ’ File to select the source file manually and import it into your solution.

In this situation, if both the source code and debug symbol files are available, you can debug using the source code. Otherwise, you can only debug by observing and setting breakpoints in the assembly code within the Disassembly window.

There are some limitations to debugging applications within the IDE, such as effective debugging of a Windows service. When faced with these limitations, you can opt to debug by attaching manually to a process that's already running, as described in the next section.

Process Debugging

You'll often encounter situations where you want to debug programs that you can't, or don't wish to, load into the IDE. Some examples of these situations include the following:

• Debugging a program that was not created with Visual Studio, such as a VB 6.0 component

• Debugging a production component that's already running

• Debugging any program running on another machine, such as a Web service

• Debugging a program running in another process, such as an ISAPI DLL

• Debugging a program difficult to run in the IDE, such as a Windows service

In these cases you need to attach the debugger manually to the process or processes that you want to debug. To attach to a process, you simply tell the debugger which process you wish to debug and the debugger will jump on its back, in a manner of speaking. Once the debugger is attached, you can break into the process and debug it just as though it was running within the IDE. After you've finished debugging the process, you tell the debugger to detach from it and the debugger will climb off its back.

Attaching to a Process

You can find the dialog window that controls attaching to, and detaching from, a process by selecting the Processes menu item on the Debug menu. Understanding the subtleties of this dialog window is important because you are likely to use it frequently, and because you should treat attaching to a process and detaching from it with some care. The Processes dialog window is shown in Figure 3-2.

Figure 3-2: The Processes dialog window

The Available Processes subwindow shows the processes executing on the machine specified in the Name combo box. If you choose the option to show the system processes, you can view processes running under the system account, such as Windows services. The Refresh button is useful because this window is not updated in real time and therefore may not reflect the true state of processes that have started and stopped while the window was showing.

The Transport combo box controls the transport protocol used for remote debugging. By default, the Visual Studio debugger uses the Distributed Component Object Model (DCOM) protocol for remote debugging. If DCOM is not available for some reason, and then only if you're debugging native Win32 applications, you can specify the Native-only TCP/IP transport. If you've upgraded to Visual Studio 2003, you have the additional option of using the Native-only Pipe transport protocol, which is more secure than TCP/IP, but slower. You use the Name combo box to specify the machine that you wish to debug remotely. Choosing a valid machine name shows the processes running on that machine. Please see Chapter 15 for more information about these options and about remote debugging.

When you click on a process in the Available Processes subwindow, the debugger enables the Attach button. Clicking this button prompts the debugger to ask you which types of application you wish to debug. A process can be running up to four different types of code, so you can choose which types of debugging you wish to perform. Figure 3-3 shows this dialog window.

Figure 3-3: The Attach to Process dialog window

The Common Language Runtime option is for debugging applications built using .NET. The Microsoft T-SQL option is for SQL Server stored procedures and triggers. The Native debugger option is for native Win32 applications, and the Script debugger option works for debugging VBScript or JScript pages. A neat shortcut is to Ctrl-click the Attach button. Provided that you have native debugging installed, this bypasses the debugger selection page and uses just the native debugger without asking.

Attaching the Visual Studio debugger to a process can lead to issues, especially when you're debugging remotely. This means that it isn't uncommon to see errors during the attach phase. Typically this happens when the debugging components necessary for one of the program types haven't been installed or configured correctly. This can cause the debugger to fail in attaching to the related program type. When this happens, you'll see a message explaining which program types the debugger attached to successfully and which program types caused the debugger attach to fail.

As long as the debugger attaches to a single program type in a process, it's successfully attached to that process. The problem is that you won't be able to debug the other program types running within that process. If you wish to receive a more detailed error message about an attach error, you should detach from the partially attached process and then attempt to attach to the process again, but only using the program type that failed. You should then see a better explanation of what went wrong with attaching to that program type.

If the debugger completely fails to attach, you'll also see an error message ” for example "Access denied " because of a permissions problem. In this case, it isn't uncommon for the process to be shown as attached even though it isn't. Clicking the Refresh button will show you the real status of the process.

For a more detailed understanding of the debugging components and setup required for each of these program types, you should read the chapter appropriate for each of the program types in which you are interested.

Breaking into a Process

You can break into any of the processes being debugged by clicking the Break button on the Processes dialog window. This is the manual equivalent of setting a breakpoint and performing a program action that causes the breakpoint to be triggered. The major difference is that with the Break button you have no control over where the program will be halted.

By default, the Break button breaks into all of the processes being debugged simultaneously. To change this behavior, go to Tools ’ Options ’ Debugging and select the option marked "In break mode, only stop execution of the current process". This is useful when you want to investigate a single process, but keep the other processes in your application running normally.

Detaching from a Process

Depending on the type of processes that you are debugging and the version of Windows that you are running, you can stop debugging your solution either automatically or manually. To stop debugging automatically, you simply select the Stop Debugging option on the Debug menu. If you want to stop debugging manually, you can use either the Detach or the Terminate button on individual processes shown in the Processes window.

The automatic option is the easiest , but it has a major annoyance. By default, Visual Studio always terminates every process from which it's detaching. This may be okay if you're only running processes from within Visual Studio, but it's nasty if you're debugging a production process or any other process that you want to keep running. To change this behavior, you have to go to the Processes window, choose the processes that you want to keep running, and for each one select the Detach from Process option rather than the default Terminate process option. You need to do this for every process and for every debug session, which rapidly turns into a major pain.

You can stop debugging a process manually by using the Detach button in the Processes window. If you're using Windows XP, this button is always enabled, and using it stops debugging without any harm to the process from which you're detaching. If, however, you're using Windows 2000 or Windows NT, and you're debugging a native Win32 process (one not created using .NET technology), the Detach button may not be enabled and you then have to use the Terminate button instead. This has the unfortunate property of doing exactly what its name implies: It kills the process that you've just been debugging. This isn't always what you want to happen. For obvious reasons, you should be especially careful about terminating a production process, a system process, or any process that other users are likely to be depending on.

This process termination happens because Win32 processes were not really designed to cohabit with a debugger; if you've ever used the Visual C++ debugger, you'll know that it never gave you the choice of detaching from a process without terminating it.

The key to fixing this problem with Win32 processes under Windows 2000 or Windows NT is to find and start the Visual Studio Debugger Proxy Windows service before debugging. This service then sits between the Visual Studio debugger and the native Win32 process being debugged, and allows you to detach from the process without terminating it. The debugger proxy service does have a few restrictions. It won't work on Windows 98 or Windows ME, and also doesn't work when you're debugging COM Interop applications. Finally, you can't use the debugger proxy service when you're debugging under Terminal Server. Whether or not you're using the proxy debugger service, Visual Studio doesn't allow you to detach from a process if you're debugging more than one project type within that process. For instance, if you're debugging a Web service together with a non-Web client of the Web service, the option to detach won't be available. I don't know why this restriction exists.

JIT Debugging

Just-In-Time (JIT) debugging shouldn't be confused with JIT compiling. It is instead an exotic variant of the process debugging discussed previously. It's designed to attach a debugger at the point that a managed or unmanaged program crashes due to an unhandled error. Without the presence of a JIT debugger, a crashed program simply closes down and vanishes.

JIT Debugger Settings with Managed Code

For a crash in managed code, and depending on the JIT debugger settings, the CLR can perform one of the following actions:

• Terminate the program after displaying some diagnostic information.

• Launch a debugger and break at the point where the crash occurred.

• Give the user a choice between the two preceding options.

This gives you some choices about handling a program crash. On a developer's machine, you probably want to use the second or third option. On an end user's machine, a better choice would probably be the first option.

The two registry settings that control this behavior are both under HKEY_LOCAL_ MACHINE, abbreviated as HKLM. The first registry setting controls how the crash is handled. The second registry setting specifies which debugger to launch if one is needed. Both settings and their possible values are shown in Table 3-3.

Table 3-3: Registry Settings for JIT Debugging of Managed Code

REGISTRY SETTING	POSSIBLE VALUES
HKLM\Software\Microsoft\ .NetFramework\ DbgJITDebugLaunchSetting	0: Show a message giving the user a choice of OK (dumps the call stack and terminates the process) or CANCEL (launches the listed debugger). 1: Dump the call stack and terminate the process. 2: Launch the debugger that is specified under the DbgManagedDebugger registry setting.
HKLM\Software\Microsoft\ .NetFramework\ DbgManagedDebugger	The default debugger is cordbg.exe, the SDK command-line debugger. You can change this value to dbgclr .exe to invoke the SDK graphical debugger or devenv /debugexe to launch the Visual Studio debugger.

JIT Debugger Settings with Unmanaged Code

For a crash in unmanaged code, you have fewer options. The two choices are as follows :

• Launch a debugger and break at the point where the crash occurred.

• Give the user a choice of launching a debugger or terminating the crashed program.

On a developer's machine you can use either option, but on an end user's machine a better choice would probably be the second option.

Once again, the two registry settings that control this behavior are both under HKEY_LOCAL_MACHINE. The first registry setting controls how the crash is handled. The second registry setting specifies which debugger to launch if one is needed. Both registry settings and their possible values are shown in Table 3-4.

Table 3-4: Registry Settings for JIT Debugging of Unmanaged Code

REGISTRY SETTING	POSSIBLE VALUES
HKLM\Software\Microsoft\Windows NT\ Current Version\AeDebug\Auto	0: Show a message giving the user a choice of OK (terminates the process) or CANCEL (launches the listed debugger) 1: Launch the debugger that is specified under the Debugger registry setting.
HKLM\ Software\Microsoft\Windows NT\ CurrentVersion\AeDebug\Debugger	The default debugger is Dr. Watson, and the default behavior is to show a message giving the user a choice (in other words, Auto is set to 0).

An alternative way of managing these settings is from within Visual Studio. Here you can set the Visual Studio debugger to handle crashes in managed code, native code, and even script by going to Tools ’ Options ’ Debugging ’ Just-In-Time and setting the appropriate options. If you didn't install the Visual C++ .NET section of Visual Studio, you may not see the native code switch in this dialog window. This is, in my opinion, a bug because you may still want to JIT-debug native code such as VB.Classic even if you're ignoring C++. If you fell into this trap, you should either install Visual C++ .NET or use the registry settings mentioned previously.