Install the Operating System Symbols and Set Up a Symbol Store
|
As anyone who has spent more than 5 minutes of development time on Windows knows, getting the correct symbols lined up is the secret to debugging faster. In managed code, if you don't have symbols, you might not be able to debug at all. In native code, without symbols, you probably won't get clean call stacks because to walk stacks, you need the frame pointer omission (FPO) data that's included as part of the PDB file.
If you think you have trouble getting everyone on your team and in the company to work with the correct symbols, think about how bad the operating system team at Microsoft has it. They have the largest commercial application in the world, with 40 plus million lines of code. They build every day and can have thousands of different builds of the operating system running at any time across the world. All of a sudden, your symbol challenges seem quite small—even if you think you're on a big project, your project can't even hold a candle to that much symbol pain!
In addition to the challenge of getting the symbols lined up, Microsoft was also facing the problem of getting the binaries lined up. One new technology that Microsoft introduced to help them debug crashes better is called a minidump, or a crash dump. These are files that contain the state of the application at the time of the crash. For some of you coming from other operating systems, you might refer to these as core dumps. The beauty of a minidump is that by having the state of the application, you can load it up into the debugger so it's almost as if you were sitting there at the time of the crash. I'll discuss the mechanics of creating your own minidumps, as well as how to read them in the debuggers, in subsequent chapters. The big issue with minidumps is getting the correct binaries loaded. While you might be developing on a post–Windows Server 2003 operating system, the customer's minidump could have been written on Windows 2000 with only Service Pack 1 applied. Like the case with the symbols, if you can't get the exact binaries loaded that were in the address space when the minidump was written, you're completely out of luck if you think you can solve the bug easily with the minidump.
The developers at Microsoft realized they had to do something to make their lives easier. We folks outside Microsoft also had been complaining for a long time that our debugging experiences were a few steps short of abysmal because of the lack of operating system symbols and binaries that matched the myriad of hot fixes on any machine. The concept of symbol servers is simple: store all the public builds symbols and binaries in a known location, and make the debuggers smarter so that they load the correct symbols and binaries for every module loaded into a process—regardless of whether that module is loaded from your code or from the operating system—without any user interaction at all. The beauty is that the reality is nearly this simple! There are a few small issues that I'll point out in this section, but with the symbol server properly set up, no one on your team or in your company should ever lack the correct symbols or binaries regardless of whether you're doing managed, native, or both styles of development, or you're using Visual Studio .NET or WinDBG as your debugger. Even better, I've supplied a couple of files that will take all the thinking work out of ensuring you have the perfect symbols and binaries for the operating system as well as for your products.
The Visual Studio .NET documentation mentions a technique for getting the symbol server set up for debugging, but you'll have to follow the same steps for every solution you load, which is a huge pain. Additionally, the documentation doesn't discuss the most important idea, which is getting your symbols and binaries into the symbol server. Since that's where the huge benefit to using the symbol server lies, you need to follow these steps to reach symbol server nirvana.
The steps for getting a server machine that everyone in your company who is executing your projects can access are quite simple. You'll probably want to name this server \\SYMBOLS to identify it easily. For the rest of this discussion, I'll assume that's the name of the server. This machine doesn't have to have much horsepower. It's simply going to act as a file server. One thing you'll definitely want is a good bit of disk space on that server. At least 40 to 80 GB should be a good start. Once the server software's installed, create two shared directories named OSSYMBOLS and PRODUCTSYMBOLS that everyone in development and QA has read and write access to. As you can tell by the share names, one directory is for the operating system symbols and binaries, and the other is for your product's symbols and binaries. You'll want to keep them separate for ease of management. Of course, it's quite easy for me to assume that you can get a server in your organization. I'll leave all the political battles of getting that server as an exercise for the reader.
The next step toward reaching developer machine symbol nirvana is to either download and install the latest version of Debugging Tools for Windows from www.microsoft.com/ddk/debugging or install Debugging Tools for Windows with this book's sample files, because the symbol server binaries are developed by the Windows team, not the Visual Studio .NET team. You'll want to check back for updated versions of Debugging Tools for Windows; the team seems to be updating the tools frequently. After installing Debugging Tools for Windows, add the installation directory to the master PATH environment variable. The four key binaries, SYMSRV.DLL, 
DBGHELP.DLL, SYMCHK.EXE, and SYMSTORE.EXE, must be able to read from and write to your symbol servers.
For those of you who are working behind proxy servers that require you to log in each time you access the Internet, you have my sympathies. Fortunately, the Windows team does feel your pain. New with Debugging Tools for Windows version 6.1.0017 is a version of SYMSRV.DLL that will work for companies that monitor your every Internet packet. You'll want to read the Debugging Tools for Windows documentation that discusses proxy and firewalls under the topic "Using Symbol Servers and Symbol Stores." In there, you'll see how to set up the _NT_SYMBOL_PROXY environment variable to download symbols without requiring you to enter your username and password with each request. You'll also want to keep an eye on www.microsoft.com/ddk/debugging for new versions of Debugging Tools for Windows. Since the Windows team is always looking for ways to make symbol servers better, you should keep an eye on new releases.
Once you have Debugging Tools for Windows installed, the final step is to set up the master environment for Visual Studio and WinDBG. It's best to set this environment variable in the master (machine-wide) settings. To access this area in Windows XP and Windows Server 2003, right-click on My Computer and select Properties from the shortcut menu. Click on the Advanced tab, and at the bottom of the property page click the Environment Variables button. Figure 2-10 shows the Environment Variables dialog box. You'll want to create a new _NT_SYMBOL_PATH environment variable, if one doesn't exist, and set it to the following. (Note that the line below is all supposed to be entered on one line.)
Figure 2-10: The Environment Variables dialog box
SRV*\\Symbols\OSSymbols*http://msdl.microsoft.com/download/symbols; SRV*\\Symbols\ProductSymbols
_NT_SYMBOL_PATH is where Visual Studio .NET and WinDBG will look to figure out where the symbol servers are. There are two distinct symbol servers being accessed in the above string separated by the semicolon. The first is for the operating system and the second is your product. The SRV in front of each tells the debuggers to load SYMSRV.DLL and to pass the values after the SRV string to SYMSRV.DLL. In the first symbol server, you are telling SYMSRV.DLL to access \\Symbols\OSSymbols as the symbol server; the second asterisk is the HTTP address where SYMSRV.DLL will look to download any symbols, but not binaries, not found in the existing symbol server. The first portion of the _NT_SYMBOL_PATH above is what will take care of getting your operating symbols completely up to date. The second portion of the _NT_SYMBOL_PATH string simply tells SYMSRV.DLL to only look in your \\Symbols\ProductSymbols share for your product specific symbols. If you want to search other paths, you can add those paths to the string _NT_SYMBOL_PATH by separating them with semicolons. For example, the following adds your system root System32 directory because that directory is where Visual Studio .NET puts the C run-time and MFC PDB files during installation:
SRV*\\Symbols\OSSymbols*http://msdl.microsoft.com/download/symbols; SRV*\\Symbols\ProductSymbols;c:\windows\system32
The absolute beauty of symbol server is revealed when you populate it with operating system symbols downloaded from Microsoft. If you've been a good bugslayer over the years, you're probably already installing the operating system symbols on your machine. However, those installations have always been a little frustrating because you probably have a few hot fixes on your machine, and certain operating system symbols never include the hot fix symbols. The great news with symbol servers is that you can be guaranteed of always getting the operating system symbols that are exactly right with no work whatsoever! This is a huge boon that will make your life easier. The magic here is that Microsoft has made the symbols available for downloading for all released operating systems from Microsoft Windows NT 4 through the latest release of Windows XP and Windows .NET Server 2003, including all operating system service packs and hot fixes.
The next time you start debugging, the debugger automatically sees that _NT_SYMBOL_PATH is set, starts downloading the operating symbols through HTTP from Microsoft, and then places them in your symbol store if the symbol file hasn't already been downloaded. Just to be clear: the symbol server will download only the symbols it needs, not every single operating system symbol. By having the symbol store in a shared directory, when one of your teammates has already downloaded the symbol, you don't have to go through the potentially long download.
The symbol store itself is nothing very exciting. It's simply a database that happens to use the file system to find the files. Figure 2-11 shows a partial listing from Windows Explorer of the tree for the symbol server on my symbol server computer. The root directory is OSSymbols, and each different symbol file, such as ADVAPI32.PDB, is listed at the first level. Under each symbol filename is a directory that corresponds to the date/time stamp, signature, and other information necessary to completely recognize a particular version of that symbol file. Keep in mind that if you have multiple versions of a file such as ADVAPI32.PDB for different operating system builds, you'll have multiple directories under ADVAPI32.PDB for each unique version you have accessed. In the signature directory, you'll most likely have the particular symbol file for that version. There are provisions for having special text files point to other locations in the symbol store, but using them the way I recommend, you'll have the actual symbol files.
Figure 2-11: An example of the symbol server database
Although getting the symbols downloaded while you're debugging is great, it does nothing for getting the operating system binaries into your symbol server. Additionally, instead of relying on developers debugging applications to get the symbols, you might want to prepopulate your symbol servers with all the operating system binaries and symbols for all versions you are supporting. That way, you'll be able to handle any minidumps coming in from the field as well as any debugging challenges you'll encounter in your development shop.
The Debugging Tools for Windows (which includes WinDBG) includes two tools that do the heavy lifting. The first, Symbol Checker (SYMCHK.EXE), takes care of getting the symbols down from Microsoft and into your symbol server. The second, Symbol Store (SYMSTORE.EXE), takes care of getting the binaries into the symbol store. Since I realized I'd have to run both tools to get my operating system symbol server fully populated with symbols and binaries for all OS versions I wanted to support, I decided to automate the process. I wanted to quickly build up my operating system symbol server as well as keep it filled with the latest binaries and symbols with essentially no work at all.
When you're setting up your initial operating system symbol server, you'll install the first version of the operating system without any service packs or hot fixes. You'll install the Debugging Tools for Windows, and you'll probably want to add its installation directory to your path. To get the binaries and symbols for that operating system, you'll run my OSSYMS.JS file, which I'll discuss in a moment. After OSSYMS.JS finishes, you'll install the first service pack and reexecute OSSYMS.JS. After you've gotten all service packs loaded and their binaries and symbols copied, you'll finally apply any hot fixes recommended by the Windows Update feature of Windows 2000, Windows XP, and Windows .NET Server 2003 and run OSSYMS.JS one last time. Once you run through this process for all operating systems you need to support, you'll just have to run OSSYMS.JS whenever you install a hot fix or a new service pack to keep your symbol server nice and spiffy. For planning purposes, I've found that it takes a little less than a gigabyte for each operating system and approximately the same for each service pack.
While you might think OSSYMS.JS (along with its support file WRITEHOTFIXES.VBS, which you'll need to copy to the same directory as OSSYMS.JS) is just a simple wrapper around calling SYMCHK.EXE and SYMSTORE.EXE, it's actually a pretty nice wrapper. If you look at the command-line options for both programs, you'll definitely want help automating them because it's very easy to mess up their usage. If you execute OSSYMS.JS without any command-line parameters, you'll see the following output showing you all the options:
OSsyms - Version 1.0 - Copyright 2002-2003 by John Robbins Debugging Applications for Microsoft .NET and Microsoft Windows Fills your symbol server with the OS binaries and symbols. Run this each time you apply a service pack/hot fix to get the perfect symbols while debugging and for mini dumps. SYMSTORE.EXE and SYMCHK.EXE must be in the path. Usage : OSsyms <symbol server> [-e|-v|-b|-s|-d] <symbol server> - The symbol server in \\server\share format. -e - Do EXEs as well as DLLs. -v - Do verbose output. -d - Debug the script. (Shows what would execute.) -b - Don't add the binaries to the symbol store. -s - Don't add the symbols to the symbol store. (Not recommended)
The only required parameter is the symbol server in \\server\share format. When you execute OSSYMS.JS, it first figures out the operating system version and service pack level as well as all hot fixes that are applied. This is important for properly filling out the product, version, and comment information for SYMSTORE.EXE so that you can identify exactly what symbols and binaries you have in your symbol server. I'll discuss the specific SYMSTORE.EXE command-line options and how to see what's in your database in a few paragraphs. Getting the hot fix information is vital because that way if you get a minidump from the field, you can quickly determine if you've got that particular binary and symbols pair in your symbol server.
Once OSSYMS.JS has all the system information, it runs through and recursively adds all DLL binaries from the operating system directory (%SYSTEMROOT%). After the binary files are copied, OSSYMS.JS calls SYMCHK.EXE to automatically download all the symbols it can for those DLLs. If you would like to also add all EXE binaries and associated symbols, add the –e command-line option to OSSYMS.JS after the symbol server parameter.
To see what binaries and symbols were added or ignored (including the reasons why), check out the two text files, DllBinLog.TXT and DllSymLog.TXT, which show the binary add results and the symbol add results, respectively. For EXEs, the two files are ExeBinLog.TXT and ExeSymLog.TXT.
Keep in mind that OSSYMS.JS can take quite a while to run. Copying the binaries up to your symbol server will be very fast, but downloading the symbols can take a lot of time. If you download both DLL and EXE operating system symbols, you've probably got somewhere around 400 MB of data to download. One thing you'll want to avoid is having multiple computers adding binary to the symbol server at once. That's because the SYMSTORE.EXE program uses the file system and a text file as its database, so it has no sort of transactional capabilities. SYMCHK.EXE doesn't access the SYMSTORE.EXE text file database, so it's perfectly fine to have multiple developers adding symbols only.
Microsoft is putting more of its product's symbols on their public HTTP download area all the time. OSSYMS.JS is flexible enough that you can easily add different directories where you'd like to have binaries as well as symbols installed into your symbol server. To add your new binaries, search for the g_AdditionalWork, a global variable near the top of the file. The OSSYMS.JS file has g_AdditionalWork set to null so it's not processed in the main routine. To add a new set of files, allocate an Array and add a SymbolsToProcess class as the element. The following code snippet shows how to add processing to add all the DLLs that appear in the Program Files directory. Note that the first element doesn't always have to be an environment variable; it could have been a specific directory, such as "e:\ Program Files." However, by using a common system environment variable, you'll avoid hard-coded drives.
var g_AdditionalWork = new Array ( new SymbolsToProcess ( "%ProgramFiles%" , // Start directory. "*.dll" , // Searching wildcard. "PFDllBinLog.TXT" , // Binary logging file. "PFDllSymLog.TXT" ) // Symbol logging file. ) ;
Now that you've seen how to get your operating system symbols and binaries, let's turn to getting your product symbols into the symbol store with SYMSTORE.EXE. SYMSTORE.EXE has a number of command-line switches. I show the important switches in Table 2-2.
| Switch | Explanation |
|---|---|
| add | Adds files to a symbol store |
| del | Deletes the files added in a particular set |
| /f File | Adds a particular file or directory |
| /r | Adds files or directories recursively |
| /s Store | The root directory of the symbol store |
| /t Product | The name of the product |
| /v Version | The product version |
| /c | Additional comment |
| /o | Verbose output helpful for debugging |
| /i ID | The ID from history.txt to delete |
| /? | Help output |
The best way to use SYMSTORE.EXE is to have it automatically add your build tree's EXEs, DLLs, and PDBs at the end of every daily build (after the smoke test verifies that the product works), after each milestone, and for any builds sent outside the engineering team. You probably don't want to have developers adding their local builds unless you're really into chewing up tons of disk space. For example, the following command stores all PDB and binary files into your symbol store for all directories found under D:\BUILD (inclusive).
symstore add /r /f d:\build\*.* /s \\Symbols\ProductSymbols /t "MyApp" /v "Build 632" /c "01/22/03 Daily Build"
Although the /t (Product) option is always required when adding files, unfortunately, /v (Version) and /c (Comment) are not. I strongly recommend that you always use /v and /c because you can never have too much information about what files are in your product symbol server. This gets extremely important as your product symbol server fills up. Even though the symbols placed in your operating system symbol server are smaller because they are stripped of all private symbols and types, your product's symbols are huge and can lead to quite a bit of wasted disk space on a six-month project.
You'll always want to leave milestone builds as well as builds sent outside the engineering team in your symbol server. I also like to keep no more than the last four weeks' daily build symbols and binaries in my symbol store. As you saw in Table 2-2, SYMSTORE.EXE does support deleting files.
To ensure that you're deleting the correct files, you'll need to look at a special directory, 000admin, under your shared symbol server directory. In there is the HISTORY.TXT file, which contains the history of all transactions that occurred in this symbol server and, if you've added files to the symbol server, a set of numbered files that contain the list of actual files added as part of a transaction.
HISTORY.TXT is a comma separated value (CSV) file whose fields are shown in Table 2-3 (for adding files) and Table 2-4 (for deleting files).
| Field | Explanation |
|---|---|
| ID | The transaction number. This is a 10-digit number, so you can have 9,999,999,999 total transactions in your symbol server. |
| Add | When adding files, this field will always say add. |
| File or Ptr | Indicates whether a file (file) or a pointer to a file in another location (ptr) was added |
| Date | The date of the transaction |
| Time | The time the transaction started |
| Product | The product text from the /t switch |
| Version | The version text from the /v switch (optional) |
| Comment | The comment text from the /c switch (optional) |
| Unused | An unused field for future use |
| Field | Explanation |
|---|---|
| ID | The transaction number |
| Del | When deleting files, this field will always say del. |
| Deleted Transaction | The 10-digit number of the deleted transaction |
Once you've located the transaction ID you want to delete, it's a simple matter to tell SYMSTORE.EXE to do the work.
symstore del /i 0000000009 /s \\Symbols\ProductSymbols
One thing I've noticed that's a little odd about deleting from your symbol server is that you don't get any output telling you if the deletion succeeded. In fact, if you forget a vital command-line option, such as the symbol server itself, you're not warned at all and you might mistakenly think that the deletion happened. After doing a deletion, I always check the HISTORY.TXT file to ensure that the deletion actually took place.
Your Source and Symbol Servers
After getting all the symbols and binaries lined up for debugging, the next piece of the puzzle is to line up your source code. Having perfect call stacks is nice, but single-stepping through comments in the source code is not fun. Unfortunately, until Microsoft integrates the compilers with the version control system so that the compilers pull and label the source as you build, you're going to have to do some manual work.
You might not have noticed it, but the compilers that come with Visual Studio .NET all now embed the complete path to the source file as part of the PDB files. Prior versions of the compilers didn't, so it was nearly impossible to get your source straight. With the complete path there, you now have a fighting chance to get the source lined up when debugging prior versions of your product or looking at a minidump.
On your build machine, use the SUBST command to set the top level of your development tree to be the S: drive. Now when you build, the S: drive will be at the root of the embedded source information for all PDB files you'll add to your symbol engine. When a developer needs to debug a prior version's source code, he can pull that version out of the version control system and use the SUBST command to map an S: drive to that pulled version of the source code. When the debuggers go to show the source code, they'll get the correct version of the symbol files with a minimum of fuss.
I've covered quite a bit about symbol servers, but I strongly encourage you to read the "Symbols" section completely in the Debugging Tools for Windows documentation. The symbol server technology is so critical to debugging better that it's in your best interest to know as much about it as possible. I hope I was able to show you the value and ways to apply it better. In fact, if you haven't already set up a symbol server, you are ordered to stop reading immediately and go set one up.
|
EAN: 2147483647
Pages: 177