Son of Strike (SOS)

There's great support for debugging dumps for native applications but not for managed applications, and although managed applications are a lot less error prone, they are much harder to debug. For example, consider those many projects that already have a considerable investment in COM+ or other native technologies. You might and want to create new .NET front-ends or components that leverage your existing COM components using COM interop. When those applications crash or hang, you're instantly in a lot of pain because it's almost impossible to hack through the assembly language and walk call stacks or even find the source and line for those .NET portions of the application.

To help you see the .NET portions of a dump or live application, some very smart people at Microsoft came up with a debugger extension called SOS, or Son of Strike. The basic documentation for SOS is in the SOS.HTM file in the <Visual Studio .NET Installation Dir>\SDK\v1.1\Tool Developers Guide\Samples\SOS directory. If you open the documentation, you'll definitely see that "basic" is the operable term here. In essence, it's the list of commands in the SOS.DLL extension and a brief bit about their usage.

If you're dealing with larger .NET systems, especially heavy ASP.NET transactions, you'll also want to download the 170-page PDF file "Production Debugging for .NET Framework Applications," from http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGrm.asp. If you want to know how to handle hung ASNET_WP.EXE processes, deal with potential .NET memory management issues, and take charge of other extreme-edge problems, this is an excellent document. The folks writing the document have definitely done their debugging of numerous live production systems and their knowledge can save you quite a bit of hassle.

With these two documentation sources, you'll get a quick discussion of the SOS commands and an extreme hardcore tricks document, but there's still something missing: just how do you get started with SOS inside WinDBG? In this section, I want to help you get a leg up on starting. My hope is that you'll get enough information out of this section to understand what's going on in the "Production Debugging for .NET Framework Applications" document. I won't be covering everything, like all the garbage collector commands, because those are covered in detail in "Production Debugging for .NET Framework Applications."

Before I start, I want to show you the easy way to get SOS.DLL loaded into WinDBG. SOS.DLL is part of the .NET Framework itself, so the trick involves getting the appropriate directories in your path so that WinDBG can easily load SOS.DLL. You need to open a MS-DOS command window and execute VSVARS32.BAT, which is located in <Visual Studio .NET Installation Dir>\Common7\Tools directory. VSVARS32.BAT gets your environment set up so that all the .NET appropriate directories are in your path.

Once you've executed VSVARS.BAT, starting WinDBG from that MS-DOS command window allows you to load SOS.DLL simply by executing .load sos from the WinDBG Command window. WinDBG always puts the last loaded extension DLL onto the top of the chain, so executing !help shows you a quick listing of all the SOS.DLL commands.

Using SOS

Probably the best way to show SOS usage is with a live example. The ExceptApp program included with this book's sample files will show you how to get started with important commands. To keep things at a manageable level, I wrote this code to simply call a few methods with local variables and finally throw an exception. I'll walk through an example of debugging EXCEPTAPP.EXE with SOS so that you can learn about the important commands for finding where you are when an application using managed code crashes or hangs. With that information, you'll be in a position to more easily apply SOS to the problems you'll encounter and understand "Production Debugging for .NET Framework Applications."

After you've compiled EXCEPTAPP.EXE and set up a MS-DOS command prompt as I described earlier, open up EXCEPTAPP.EXE in WinDBG and stop at the loader breakpoint. To make WinDBG stop when a .NET application throws an exception, you have to tell WinDBG about the exception number that .NET throws. The easiest way to do this is to go into the Event Filters dialog box, click the Add button, and in the Exception Filter dialog box, enter 0xE0434F4D. Select Enabled in the Execution group box, and select Not Handled in the Continue group box. Once you click OK, you've successfully set WinDBG to stop whenever any EXCEPTAPP.EXE throws a .NET exception. If the value 0xE0434F4D looks somewhat familiar, you can always see what it stands for by using the .formats command.

After you have the exception set, run EXCEPTAPP.EXE until it stops on the .NET exception. WinDBG will report it as a first chance exception and stop the application on the actual Win32 API RaiseException call. After getting SOS loaded with a .load sos command, execute !threads (first command you'll always want to execute in SOS) so that you can see which threads in the application or dump have .NET code in them. With EXCEPTAPP.EXE, the WinDBG thread command ~ indicates that three commands are running in the application. However, the all-important !threads command lists that only threads 0 and 2 have any .NET code in them, as shown in the following output. (To get everything to fit in on the page, I show the individual thread information in a table. In WinDBG, you see it as a long horizontal display.)

0:000> !threads PDB symbol for mscorwks.dll not loaded  succeeded Loaded Son of Strike data table version 5 from "e:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll" ThreadCount: 2 UnstartedThread: 0 BackgroundThread: 1 PendingThread: 0 DeadThread: 0    

Row Heading
WinDBG Thread ID	0	2
Win32 Thread ID	884	9dc
ThreadObj	00147c60	001631c8
State	20	1220
PreEmptive GC	Enabled	Enabled
GC Alloc Context	04a45f24:04a45ff4	00000000:00000000
Domain	00158300	00158300
Lock Count	0	0
APT	Ukn	Ukn
Exception	System.ArgumentException	(Finalizer)

The important information in the !threads display consists of the Domain field, because that tells you whether multiple AppDomains are running in this process; and the Exceptions field, which happens to be overloaded. In the EXCEPTAPP.EXE example, the first thread has thrown the System.ArgumentException, so you can see the current exception for any thread. The third thread in EXCEPTAPP.EXE shows the special value (Finalizer), which indicates the thread is, as you can guess, the finalizer thread for the process. You'll also see (Theadpool Worker), (Threadpool Completion Port), or (GC) in the Exception field. When you see one of those special values, you'll know they represent runtime threads, not your threads.

Since we've determined that the WinDBG thread 0 contains the EXCEPTAPP.EXE exception, you'll want to take a look at the call stack with !clrstack –all to see all the details about the stack, including parameters and locals. Although !clrstack has switches to see the locals (-l) and parameters (-p), if you specify them together, they seem to cancel each other out and you see neither. If you'd like to walk all thread call stacks at once, you can use the command ~*e !clrstack.

** Note, I excised the registers from this display ** 0:000> !clrstack –all Thread 0 ESP       EIP      0012f5e0  77e73887 [FRAME: HelperMethodFrame]  0012f60c  06d3025f [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Doh                                        (String,ValueClass ExceptApp.Days)    at [+0x67] [+0x16] c:\junk\cruft\exceptapp\class1.cs:14     PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)     PARAM: value class ExceptApp.Days StrParam     PARAM: unsigned int8 ValueParam: 0x07 0012f630  06d301e2 [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Reh                                                    (I4,String)   at [+0x6a] [+0x2b] c:\junk\cruft\exceptapp\class1.cs:23     PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)     PARAM: class System.String i: 0x00000042      PARAM: int8 StrParam: 77863812     LOCAL: class System.String s: 0x04a45670 (System.String)     LOCAL: value class ExceptApp.Days e: 0x003e5278 0x0012f63c  

In the parameter display, there seems to be a bug because !clrstack doesn't always display the parameter types correctly. In the DoSomething.Doh method, you can see it takes a String (StrParam) and a Days (ValueParam) value enumeration in the prototype. However, the PARAM: information shows the StrParam parameter as value class ExceptApp.Days and ValueParam as unsigned int8. Fortunately for value parameters, even when the type is wrong, the correct value displays next to the parameter name. In the ValueParam example, the value passed in is 7, which corresponds to the enumeration Fri.

Before I jump into figuring out the values of the value classes and objects, I want to mention one other stack walking command you might find useful. If you're dealing with heavy cross–.NET and native calls and you'd like to see a call stack that includes everything, the !dumpstack command is your friend. Overall, it does a good job, but it looks like having full PDB symbols for the .NET Framework would make it better. Occasionally, the !dumpstack command reports "Use alternate method which may not work", which seems to indicate that it's attempting to walk the stack when it's missing certain symbol information.

In the LOCAL: display under the call to DoSomething.Reh are two local variables: s, a String object; and e, a Days value class. After each comes the hexadecimal address describing the type. For the Days value, there are two numbers, 0x003E5278 and 0x0012F63C. The first number is the method table and the second is the location in memory for the value. Seeing the value in memory is simple using one of WinDBG's memory dumping commands such as dd 0x0012F63C.

Seeing the method table that describes the method data, the module information, and the interface map, among other things, is done through SOS's !dumpmt command. Executing !dumpmt 0x003E5278 with the EXCEPTAPP.EXE example shows the following:

0:000> !dumpmt 0x003e5278  EEClass : 06c03b1c Module : 001521a0 Name: ExceptApp.Days mdToken: 02000002  (D:\Dev\ExceptApp\bin\Debug\ExceptApp.exe) MethodTable Flags : 80000 Number of IFaces in IFaceMap : 3 Interface Map : 003e5380 Slots in VTable : 55

With the method table, in the first two numbers displayed, you can see which module a method comes from as well as its execution engine class. For interfaces, the SOS documentation has an excellent example of how to walk the interface maps, and I would encourage you to look it over. If you have a burning desire to see all the methods in the v-table for a particular class or object along with their method descriptors, you can specify the –md option in front of the method table value. In the case of EXCEPTAPP.EXE's value class ExceptApp.Days, you'll see all 55 methods listed. As the SOS documentation mentions in the "How Do I… ?" section, getting the method descriptors is important to setting breakpoints on specific methods.

Since we're looking at the class and module information for the ExceptApp.Days method table, I want to take a little detour. Once you have an execution engine class address, the !dumpclass command will show you everything you ever wanted to see about a class, with the important information being all the data fields in the class. To see the information about a module, use the !dumpmodule command. The !dumpmodule output documentation has examples of how to walk through memory and find classes and method tables for a module.

Now that we've ground through the value class, let's take a look at making sense out of the String local variable s, in DoSomething.Reh, which was displayed as follows:

LOCAL: class System.String s: 0x04a45670 (System.String)

As s is an object, only one hexadecimal value is displayed after the variable name—the location of that object in memory. Using the !dumpobj command, you'll see all the information about that object.

0:000> !dumpobj 0x04a45670  Name: System.String MethodTable 0x79b7daf0 EEClass 0x79b7de3c Size 92(0x5c) bytes mdToken: 0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Tommy can you see me? Can you see me? FieldDesc*: 79b7dea0       MT    Field   Offset         Type       Attr    Value Name 79b7daf0  4000013       4  System.Int32   instance       38 _arrayLength 79b7daf0  4000014       8  System.Int32   instance       37 m_stringLength 79b7daf0  4000015       c   System.Char   instance       54 m_firstChar 79b7daf0  4000016       0         CLASS     shared   static Empty     >> Domain:Value 00158298:04a412f8 << 79b7daf0  4000017       4         CLASS     shared   static WhitespaceChars     >> Domain:Value 00158298:04a4130c <<

As you can see from the output, some of the fields, MethodTable, EEClass, and MT (aka Method Table), can be used with commands I've previously discussed. For the field members, the !dumpobj command will show the values directly in the table for simple value types. In the String display in the preceding output, the m_stringLength value is the 37 characters currently in the string. As you'll see in a moment, for object field members, the Value field will contain the object instance, and you can use the !dumpobj command on to see the value.

The entries delineated by >> and << are showing you the domain instance and location in that domain for the static field prior to the >>. If I had multiple AppDomains in EXCEPTAPP.EXE, you'd see two domains and value information output for the static WhitespaceChars field.

Now that I've covered some of the basic commands, I want to tie them together and show how you'll look up useful data with them. With EXCEPTAPP.EXE stopped in WinDBG because of an exception, it would be nice to see what the exception is and what some of the fields are so we can see why EXCEPTAPP.EXE stopped in the middle of execution.

We know from executing the !threads command that the first thread is currently processing an exception, System.ArgumentException. If you look carefully at the output for !clrstack or !dumpstack, you'll notice that no locals or parameters that show any type of System.ArgumentException are displayed. The good news is that an excellent command, !dumpstackobjects, shows all objects currently on the stack of the current thread:

0:000> !dumpstackobjects ESP/REG  Object   Name ebx      04a45670 System.String    Tommy can you see me? Can you see me? 0012f50c 04a45f64 System.ArgumentException 0012f524 04a45f64 System.ArgumentException 0012f538 04a45f64 System.ArgumentException 0012f558 04a44bc4 System.String    Reh =  0012f55c 04a45f64 System.ArgumentException 0012f560 04a45670 System.String    Tommy can you see me? Can you see me? 0012f564 04a4431c System.Byte[] 0012f568 04a43a58 System.IO.__ConsoleStream 0012f5a0 04a45f64 System.ArgumentException  

Since the !dumpstackobjects command is wandering up the stack, you'll see some items multiple times as they are passed a parameter to multiple functions. In the preceding output, you can see multiple System.ArgumentException objects, but if you look at the object value next to each object, you'll notice they are all referring to the same object instance, 0x04A45F64.

To look at the System.ArgumentException object, I'll use the !dumpobj command. I had to wrap the Name column to get everything to fit on the page.

0:000> !dumpobj 04a45f64  Name: System.ArgumentException MethodTable 0x79b87b84 EEClass 0x79b87c0c Size 68(0x44) bytes mdToken: 02000038  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) FieldDesc*: 79b87c70       MT    Field   Offset          Type       Attr    Value Name 79b7fcd4  400001d        4         CLASS   instance 00000000 _className 79b7fcd4  400001e        8         CLASS   instance 00000000                                                             _exceptionMethod 79b7fcd4  400001f        c         CLASS   instance 00000000                                                       _exceptionMethodString 79b7fcd4  4000020       10         CLASS   instance 04a456cc _message 79b7fcd4  4000021       14         CLASS   instance 00000000                                                              _innerException 79b7fcd4  4000022       18         CLASS   instance 00000000 _helpURL 79b7fcd4  4000023       1c         CLASS   instance 00000000 _stackTrace 79b7fcd4  4000024       20         CLASS   instance 00000000                                                           _stackTraceString 79b7fcd4  4000025       24         CLASS   instance 00000000                                                     _remoteStackTraceString 79b7fcd4  4000026       2c  System.Int32   instance        0                                                           _remoteStackIndex 79b7fcd4  4000027       30  System.Int32   instance -2147024809 _HResult 79b7fcd4  4000028       28         CLASS   instance 00000000 _source 79b7fcd4  4000029       34  System.Int32   instance        0 _xptrs 79b7fcd4  400002a       38  System.Int32   instance -532459699 _xcode 79b87b84  40000d7       3c         CLASS   instance 04a45708 m_paramName

Inside an exception, the Message property is the important property. Because I can't call a method directly from WinDBG to see its value, I'll have to look at the _message field because that's where the Message property stores the actual string. Since the _message field is marked with CLASS, the hexadecimal number in the Value column is the object instance. To look at the object, I'll do another !dumpobj command to view it. As we've seen, the String object will have a special field in it, so we can see its actual value, which turns out to be the innocuous "Thowing an exception."

0:000> !dumpobj 04a456cc  Name: System.String MethodTable 0x79b7daf0 EEClass 0x79b7de3c Size 60(0x3c) bytes mdToken: 0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Thowing an exception FieldDesc*: 79b7dea0       MT    Field  Offset          Type       Attr    Value Name 79b7daf0  4000013       4  System.Int32   instance       21 m_arrayLength 79b7daf0  4000014       8  System.Int32   instance       20 m_stringLength 79b7daf0  4000015       c   System.Char   instance       54 m_firstChar 79b7daf0  4000016       0        CLASS      shared   static Empty     >> Domain:Value 00158298:04a412f8 << 79b7daf0  4000017       4         CLASS     shared   static WhitespaceChars     >> Domain:Value 00158298:04a4130c <<