Set Up a Symbol Store
| As anyone who has spent more than 5 minutes of development time on Windows knows, getting the correct symbols lined up is the secret to debugging faster. Whereas managed code has only the source and line information and local variable names in the PDB file, if you have problems where native code is involved, you may be completely out of luck. Native code puts much more information into a PDB. The most important piece is the frame pointer omission (FPO) data. That's the information that allows you to walk native stacks back into your managed code. Without FPO, you are in serious trouble when it comes to debugging. If you think you have trouble getting everyone on your team and in the company to work with the correct symbols, think about how bad the operating system team at Microsoft has it. They build every day and can have thousands of different builds of the operating system running at any time across the world. Suddenly, your symbol challenges seem quite smalleven if you think you're on a big project, your project is nothing compared to that much symbol pain! In addition to the challenge of getting the symbols lined up, Microsoft was also facing the problem of getting the binaries lined up. One technology that Microsoft introduced to help them debug crashes better is called a minidump, or a crash dump. These are files that contain the state of the application at the time of the crash. For some of you coming from other operating systems, you might refer to these as core dumps. The beauty of a minidump is that by having the state of the application, you can load it up into the debugger so it's almost as if you were sitting there at the time of the crash. I'll discuss the mechanics of creating your own minidumps, in addition to how to read them in the debuggers, in subsequent chapters. The big issue with minidumps is getting the correct binaries loaded. You might be developing on a postWindows Vista operating system, but the customer's minidump could have been written on Windows Server 2003 with only Service Pack 1 applied. Like the case with the symbols, if you can't get the exact binaries loaded that were in the address space when the minidump was written, you're completely out of luck if you think you can solve the bug easily with the minidump. The developers at Microsoft realized they had to do something to make their lives easier. We folks outside Microsoft also had been complaining for a long time that our debugging experiences were a few steps short of abysmal because of the lack of operating system symbols and binaries that matched the myriad of hot fixes on any machine. The concept of Symbol Servers is simple: store all the public builds symbols and binaries in a known location, and make the debuggers smarter so that they load the correct symbols and binaries for every module loaded into a processregardless of whether that module is loaded from your code or from the operating systemwithout any user interaction at all. The beauty is that the reality is nearly this simple! There are a few small issues that I'll point out in this section, but with the Symbol Server properly set up, no one on your team or in your company should ever lack the correct symbols or binaries regardless of whether you're doing managed, native, or both styles of development, or you're using Visual Studio or WinDBG as your debugger. Even better, I've supplied a tool that will take all the thinking work out of ensuring that you have the perfect symbols and binaries for the operating system and for your products. Compared to previous versions, Visual Studio 2005 introduces a much improved way of accessing Symbol Servers for debugging. Sadly, the documentation still doesn't discuss the most important idea, which is getting your symbols and binaries into the Symbol Server. Because that's where the huge benefit to using the Symbol Server lies, I next describe how to set up an effective Symbol Server. The steps for getting a server machine that everyone in your company who is executing your projects can access are quite simple. You'll probably want to name this server \\SYMBOLS to identify it easily. For the rest of this discussion, I'll assume that's the name of the server. This machine doesn't have to have much horsepower; it's simply going to act as a file server. One thing you'll definitely want is a lot of disk space on that server. At least 80 GB should be a good start. Once the server software's installed, create two shared directories named OSSYMBOLS and PRODUCTSYMBOLS. Allow everyone in development read and write access to OSSYMBOLS. For PRODUCTSYMBOLS, developers and QA need only read access, but the account that does your builds needs write access. As you can tell by the share names, one directory is for the operating system symbols and binaries, and the other is for your product's symbols and binaries. You'll want to keep them separate for ease of management. Of course, it's quite easy for me to assume that you can get a server in your organization. I'll leave all the political battles of getting that server as an exercise for the reader. The next step is to either download and install the latest version of Debugging Tools for Windows from http://www.microsoft.com/whdc/devtools/debugging/default.mspx, because the Symbol Server binaries are developed by the Windows team, not the Visual Studio team. Make sure to install the correct version matching the "bitness" of the operating system you're using in order to debug applications correctly. You'll want to check back for updated versions of Debugging Tools for Windows; the team seems to update the tools frequently. After installing Debugging Tools for Windows, add the installation directory to the system's PATH environment variable. The four key binaries, Symsrv.dll, Dbghelp.dll, Symchk.exe, and Symstore.exe, must be able to read from and write to your Symbol Servers. For those of you who are working behind proxy servers that require you to log in each time you access the Internet, you have my sympathies. Fortunately, the Windows team does feel your pain. Debugging Tools for Windows and Visual Studio 2005 ship with a version of Symsrv.dll that will work for companies that monitor all your Internet packets. You'll want to read the Debugging Tools for Windows documentation that discusses proxy and firewalls under the topic, "Using Symbol Servers and Symbol Stores." In there, you'll see how to set up the _NT_SYMBOL_PROXY environment variable to download symbols without requiring you to type your user name and password with each request. You'll also want to look for new versions of Debugging Tools for Windows. Since the Windows team is always improving the Symbol Server binaries, you should look for new releases. In most cases, the team releases new versions in January and July. However, the team has snuck in releases at other times. Read the Wintellect Blog at http://www.wintellect.com/WEBLOGS/ because I always try to alert everyone when the team ships a new Debugging Tools for Windows. Once you have Debugging Tools for Windows installed, it's time to set up both debuggers to use it. For WinDBG, it's best to use two environment variables in the system settings to tell the debugger where the symbol servers reside. To access this area in Windows XP and Windows Server 2003, right-click My Computer and select Properties from the shortcut menu. Click the Advanced tab, and at the bottom of the System Properties dialog box, click Environment Variables. Figure 2-5 shows the Environment Variables dialog box. You'll want to create two new environment variables, _NT_SYMBOL_PATH and _NT_EXECUTABLE_IMAGE_PATH, and I'll show you the values in a moment. As you can guess from the names, the first environment variable is where WinDBG will look up PDB files, and WinDBG uses the second to look up the binaries from minidumps. The value for _NT_SYMBOL_PATH is as follows (note that the following code is all supposed to be entered on one line, and you'll put your path to the Visual Studio installation directory in place of Visual_Studio_installation_directory). SRV*c:\symbols*\\Symbols\OSSymbols*\\Symbols\ProductSymbols* http://msdl.microsoft.com/download/symbols; <VS install directory>\SDK\v2.0\symbols; <VS install directory>\SDK \v2.0 64bit\symbols; Figure 2-5. The Environment Variables dialog box
The value for _NT_EXECUTABLE_IMAGE_PATH is very similar: SRV*c:\symbols*\\Symbols\OSSymbols*\\Symbols\ProductSymbols To explain the details of the symbol path, it's good to learn how a debugger goes about looking up symbols. The key values are all in the PDB file and the GUID. When you compile source code to a Portable Executable (PE) file and produce a PDB file, the compiler/linker puts in what's called a debug section. In that section, it lists the complete path to the PDB file built in addition to a GUID. That same GUID is also written to the PDB file. By having the unique GUID in both places, the debugger does not have to rely on error-prone timestamps to ensure that it's looking at the matching PDB file for a binary. When the debugger goes to load a PDB, the first place it looks is in the debug section of the binary for the full path to the PDB file built. If the matching PDB file is there, the debugger opens that PDB file, and you're debugging with symbols. If you've ever wondered why you can build a binary on a machine and move the binary to a different drive and directory, but the debugger seems to magically know where to find symbols, this is why. If the debugger can't find the matching PDB file from the path embedded in the binary, it looks in the directory where the binary was loaded. If the matching symbols are there, they are loaded and life is good. Now you know why you can copy a binary and PDB file to a different machine and symbols properly show up. If the PDB file is not in the binary load directory, and you have the _NT_SYMBOL_PATH environment variable set, your symbol search embarks on a slightly different quest. Originally, _NT_SYMBOL_PATH was intended to contain the list of paths to search separated by semicolons. The debugger would parse out the paths and look for the matching name and GUID and if found, that PDB file was loaded. When the Symbol Server sees the path start with SRV, that's the signal to start working the symbol-finding magic. By the way, if you set the environment variable and you are still not getting any Symbol Server support, I can almost guarantee that you have swapped the R and the V in SRV. The first asterisk (*) following the SRV is the spot called the download cache. If the debugger looks through the various Symbol Servers you've set, and if it finds the PDB file, it copies the file to your download cache. When the debugger looks there and finds the matching PDB file, it will open that one and stop looking through the rest of the Symbol Server path. The download cache means that you don't experience the network performance hit of accessing the PDB file over the network every time you debug. If you're running out of disk space, all you have to do is delete everything in the C:\SYMBOLS directory the next time you debug, and the debugger will automatically repopulate with the matching PDB files. If the PDB file is not found in the download cache, the debugger looks for the Symbol Servers delineated by subsequent asterisks in the environment variable. In the value for _NT_SYMBOL_PATH above, the first place searched will be your operating system Symbol Server, \\Symbols\OSSymbols. A little later in this section, I'll show you how to populate your operating system Symbol Server with a minimum of pain. If the PDB file is found in your operating system Symbol Server, the PDB file is copied to the download cache, so you aren't required to set up the network access on the matching PDB file again. As you can guess, the \\Symbols\ProductSymbols Symbol Server contains your product symbols. You'll also learn how to get all your builds into your product Symbol Server in this chapter so you never go without symbols again. The last Symbol Server specified, http://msdl.microsoft.com/download/symbols, is the most interesting of all. If the debugger can't find the symbols in your company's Symbol Servers, it will ask Microsoft's public Symbol Server for the matching PDB file. Why this is so wonderful is that for all versions of Windows operating systems from Microsoft Windows 2000 forward, including all service packs and hot fixes, you'll get the operating system symbols automatically. If you come into work on Wednesday after "Patch Tuesday," which is the second Tuesday of the month when Microsoft releases the latest hot fixes, you'll immediately get all the updated operating system symbols as a matter of course. If you can't guess, this is a huge boon for productivity, and being able to walk the stack out of the middle of native code is fantastic. The symbol download applies regardless if you're debugging your unit test on your machine or on a minidump from a customer. The right PDB files just appear. In the case of your company Symbol Servers set up on \\SYMBOLS, you will have the PDB files in addition to the appropriate binaries necessary for minidump debugging. The Microsoft public Symbol Server has only the PDB files on it. That still might cause some issues on certain minidumps, but the debuggers will go to heroic efforts to allow the debugging to work. The last point about the public Symbol Server is that the symbol files up there are not the full native PDB files, but they're stripped so they contain only public functions and the all-important FPO data to walk the stacks. After all the Symbol Servers in the environment variable value shown earlier come two semicolon-deliniated paths: Visual_Studio_installation_directory\SDK\v2.0\symbols and Visual_Studio_installation_directory \SDK\v2.0 64bit\symbols. If you believe me that your Symbol Servers combined with the Microsoft Public Symbol Server are awesome, you might be wondering why I'm recommending that you add the Framework SDK symbols directories to your path. In most cases, the Visual Studio installation automatically installs the Framework SDK, though you can opt not to install it, which I would not recommend. The Framework SDK symbols directory has the PDB files for the core DLLs from the Common Language Runtime (CLR). A few of the Son of Strike (SOS) WinDBG extension commands that you'll be using for the most hard-core problems need access to the PDB files to do their work. In the rare situation in which you don't have the files already in your cache and you lose your connection to the Internet, having those paths in your environment variables can make the difference when it comes to solving the problem. Now that you know about the steps that the debugger goes through for symbols, you can see the same process for the _NT_EXECUTABLE_IMAGE_PATH environment variable for binaries. Because the binary files are not in the Microsoft Public Symbol Server, you're telling WinDBG to search only your internal Symbol Servers. The one difference is that WinDBG will look for the binaries on your machine first. For example, if the minidump has a record of a binary in the C:\Foo directory, that's where WinDBG will look first before going through the steps of searching the Symbol Server. In WinDBG, it can be a bit difficult to see if you have the _NT_SYMBOL_PATH set correctly. Because this is a WinDBG issue, in the "Symbol Server Setup" section in Chapter 6, "WinDBG, SOS, and ADPlus," I go over the steps necessary for ensuring that WinDBG is properly loading symbols. Visual Studio 2005 will use the _NT_SYMBOL_PATH and _NT_EXECUTABLE_IMAGE_PATH environment variables, but you can set the exact same values in the IDE. What's even better about the way Visual Studio sets the symbol and binary paths is that once you set it, you can export and import those settings to other installations very easily. It is a good idea to have a team-wide set of settings that everyone can import when setting up a new machine and immediately be ready to start debugging. In Visual Studio 2005, in the Options dialog box, click the Debugging node and then the Symbols node. The property page there is where you'll type the paths directly. In the setup I'm advocating, you'd type the paths in the following order:
Additionally, you will set the cache directory to C:\SYMBOLS to share the cache file between the two debuggers. Figure 2-6 shows the Options dialog box with all the values properly filled out. Figure 2-6. The Visual Studio debugging symbols setup What makes the Visual Studio method superior to the WinDBG environment variable approach is that Visual Studio automatically handles Symbol Servers just by putting the \\server_name\share_name in the symbols setup. This is a major improvement compared to previous versions and will make your debugging considerably easier. The symbol store itself is nothing very exciting. It's simply a database that happens to use the file system to find the files. Figure 2-7 shows a partial listing from Windows Explorer of the tree for the Symbol Server on my Symbol Server computer. The root directory is OSSymbols, and each different symbol file, such as Advapi32.pdb, is listed at the first level. Under each symbol's file name is a directory that corresponds to the GUID to recognize a particular version of that symbol file. Keep in mind that if you have multiple versions of a file, such as Advapi32.pdb for different operating system builds, you'll have multiple directories under Advapi32.pdb for each unique version you have accessed. In the signature directory, you'll most likely have the particular symbol file for that version. Figure 2-7 shows two GUID values under the Advapi32.pdb directory. If you happen to see a directory name that's shorter than a GUID, which will be the timestamp that will indicate that you were debugging symbols from Windows 2000. Compilers prior to Visual Studio 2005 did not embed a GUID, so they used the Portable Executable (PE) file checksum as the unique identifying characteristic. Figure 2-7. An example of the Symbol Server database Although getting the symbols downloaded while you're debugging is great, it does nothing for getting the operating system binaries into your Symbol Server. Additionally, instead of relying on developers' debugging applications to get the symbols, you might want to pre-populate your Symbol Servers with all the operating system binaries and symbols for all versions you are supporting. That way, you'll be able to handle any minidumps coming in from the field in addition to any debugging challenges you'll encounter in your development shop. The Debugging Tools for Windows (which includes WinDBG) includes two tools that do the bulk of the work. The first, Symbol Checker (Symchk.exe), takes care of getting the symbols from Microsoft into your Symbol Server. The second, Symbol Store (Symstore.exe), takes care of getting the binaries into the symbol store. Since I realized that I'd have to run both tools to get my operating system Symbol Server fully populated with symbols and binaries for all OS versions I wanted to support, I decided to automate the process. I wanted to quickly build up my operating system Symbol Server and keep it filled with the latest binaries and symbols with essentially no work at all. When you're setting up your initial operating system Symbol Server, you'll install the first version of the operating system without any service packs or hot fixes. You'll install the Debugging Tools for Windows, and you'll probably want to add its installation directory to your path. To get the binaries and symbols for that operating system, you'll run my Ossyms.js file, which I'll discuss in a moment. After Ossyms.js finishes, you'll install the first service pack and reexecute Ossyms.js. After you've gotten all service packs loaded and their binaries and symbols copied, you'll finally apply any hot fixes recommended by the Windows Update feature of Windows XP, Windows Vista, and Windows Server 2003 and run Ossyms.js one last time. Once you run through this process for all operating systems you need to support, you'll just have to run Ossyms.js whenever you install a hot fix or a new service pack to keep your Symbol Server up to date. For planning purposes, I've found that it takes a little less than a gigabyte for each operating system and approximately the same for each service pack. Before you run Ossyms.js, you'll want to change the program that executes Windows Script Hosting files, from the default Wscript.exe to Cscript.exe. Wscript.exe will do all output through message boxes, and with scripts such as Ossyms.js, or any other script for that matter, you'll go nuts clicking OK. Open a command prompt with administrator privileges and execute the following command to set Cscript.exe as the Windows Script Hosting executing program: cscript //H:CScript to set While you might think that Ossyms.js is just a simple wrapper around executing Symchk.exe and Symstore.exe, it's actually a pretty nice wrapper. If you look at the command-line options for both programs, you'll definitely want help automating them because it's very easy to mess up their usage. If you execute Ossyms.js without any command-line parameters, you'll see the following output showing you all the options: OSsyms - Version 2.0 - Copyright 2002-2006 by John Robbins Debugging Microsoft .NET 2.0 Applications Fills your Symbol Server with the OS binaries and symbols. Run this each time you apply a service pack/hotfix to get perfect symbols while debugging and for minidumps. SYMSTORE.EXE and SYMCHK.EXE must be in the path. Usage : OSsyms <Symbol Server> [-e|-b|-s|-d] <Symbol Server> - Symbol server in \\server\share or drive:\dir format. -e - Do EXEs as well as DLLs. -d - Debug the script. (Shows what would execute.) -b - Don't add the binaries to the symbol store. -s - Don't add the symbols to the symbol store. (Not recommended) The only required parameter is the Symbol Server in \\server\share format. Ossyms.js runs through and recursively adds all DLL binaries from the operating system directory (%SYSTEMROOT%). After the binary files are copied, Ossyms.js calls Symchk.exe to automatically download all the symbols it can for those DLLs. If you would like to also add all EXE binaries and associated symbols, add the -e command-line option to Ossyms.js after the Symbol Server parameter. To see what binaries and symbols were added or ignored (including the reasons why), check out the two text files, DllBinLog.txt and DllSymLog.txt, which show the binary add results and the symbol add results, respectively. For EXE files, the two files are ExeBinLog.txt and ExeSymLog.txt. Keep in mind that Ossyms.js can take quite awhile to run. Copying the binaries onto your Symbol Server will be very fast, but downloading the symbols can take a lot of time. If you download both DLL and EXE operating system symbols, you probably have somewhere around 400 MB of data to download. One thing you'll want to avoid is having multiple computers adding binaries to the Symbol Server simultaneously. That's because the Symstore.exe program uses the file system and a text file as its database, so it has no transactional capabilities. Symchk.exe doesn't access the Symstore.exe text file database, so it's perfectly fine to have multiple developers adding symbols only. Microsoft is putting more of its products' symbols on their public symbol server all the time. Ossyms.js is flexible enough that you can easily add different directories where you'd like to have binaries in addition to symbols installed into your Symbol Server. To add your new binaries, search for the g_AdditionalWork, a global variable near the top of the file. The Ossyms.js file has g_AdditionalWork set to null so it's not processed in the main routine. To add a new set of files, allocate an Array type, and add a SymbolsToProcess class as the element. The following code snippet shows how to add processing to add all the DLLs that appear in the Program Files directory. Note that the first element isn't necessarily required to be an environment variable; it could have been a specific directory, such as "C:\ Program Files." However, by using a common system environment variable, you'll avoid hard-coded drives. var g_AdditionalWork = new Array ( new SymbolsToProcess ( "%ProgramFiles%" , // Start directory. "*.dll" , // Searching wildcard. "PFDllBinLog.TXT" , // Binary logging file. "PFDllSymLog.TXT" ) // Symbol logging file. ) ; Now that you've seen how to get your operating system symbols and binaries, let's turn to getting your product symbols into the symbol store with Symstore.exe. Symstore.exe has a number of command-line switches. I show the important switches in Table 2-2.
The best way to use Symstore.exe is to have it automatically add your build tree's EXEs, DLLs, and PDBs at the end of every daily build (after the smoke test verifies that the product works), after each milestone, and for any builds sent outside the engineering team. You probably don't want to have developers adding their local builds unless you're really into using up tons of disk space. For example, the following command stores all PDB and binary files in your symbol store for all directories found under D:\BUILD (inclusive): symstore add /r /f d:\build\*.* /s \\Symbols\ProductSymbols /t "MyApp" /v "Build 632" /c "01/22/06 Daily Build" Although the /t (Product) option is always required when adding files, unfortunately, /v (Version) and /c (Comment) are not. I strongly recommend that you always use /v and /c because you can never have too much information about what files are in your product Symbol Server. This becomes extremely important as your product Symbol Server fills up. Even though the symbols placed in your operating system Symbol Server are smaller because they are stripped of all private symbols and types, your product's symbols are larger and can lead to quite a bit of wasted disk space on a six-month project. You'll always want to leave milestone builds and builds sent outside the engineering team in your Symbol Server. I also like to keep no more than the last four weeks' daily build symbols and binaries in my symbol store. As you saw in Table 2-2, Symstore.exe does support deleting files. To ensure that you're deleting the correct files, you'll need to look at a special directory, 000admin, under your shared Symbol Server directory. In there is the History.txt file, which contains the history of all transactions that occurred in this Symbol Server and, if you've added files to the Symbol Server, a set of numbered files that contain the list of actual files added as part of a transaction. History.txt is a comma separated value (CSV) file whose fields are shown in Table 2-3 (for adding files) and Table 2-4 (for deleting files).
Once you've located the transaction ID you want to delete, it's a simple matter to tell Symstore.exe to do the work. symstore del /i 0000000009 /s \\Symbols\ProductSymbols One thing I've noticed that's a little odd about deleting from your Symbol Server is that you don't get any output telling you if the deletion succeeded. In fact, if you forget a vital command-line option, such as the Symbol Server itself, you're not warned at all and you might mistakenly think that the deletion happened. After doing a deletion, I always check the History.txt file to ensure that the deletion actually took place. If you're looking for a nice project, what you might consider doing is writing a form-based application to manage the symbols in your symbol store. Manually looking to see what's in the server in addition to deleting old versions of your product symbols is rather tedious. By showing the dates the files were added in addition to showing the names of the files added, you could make the delete a one-click option. |
EAN: 2147483647
Pages: 99
- Challenging the Unpredictable: Changeable Order Management Systems
- Enterprise Application Integration: New Solutions for a Solved Problem or a Challenging Research Field?
- The Effects of an Enterprise Resource Planning System (ERP) Implementation on Job Characteristics – A Study using the Hackman and Oldham Job Characteristics Model
- Distributed Data Warehouse for Geo-spatial Services
- Data Mining for Business Process Reengineering