Other GDB Debugging Topics

In this section, we ll discuss some other topics of GDB, such as multiprocess application debugging and post-mortem debugging.

Multiprocess Application Debugging

One problem with the debugging of multiprocess applications is which process to follow when a new process is created. Recall from Chapter 12, Introduction to Sockets Programming, that the fork function returns to both the parent and child processes. We can tell GDB which to returns to follow using the follow-fork-mode command. For example, if we wanted to debug the child process, we d specify to follow the child process as:

 set follow-fork-mode child 

Or, if we instead wanted to follow the parent (the default mode), we d specify this as:

 set follow-fork-mode parent 

In either case, when GDB follows one process, the other process (child or parent) continues to run unimpeded. We can also tell GDB to ask us which process to follow when a fork occurs, as:

 set follow-fork-mode ask 

When the fork occurs, GDB will ask which to follow. Whichever is not followed will execute normally.

Multithreaded Application Debugging

There s no other way to put it: Debugging multithreaded applications is difficult at best. GDB offers some capabilities that assist in multithreaded debugging, and we ll look at those here.

The breakpoint is one of the most important aspects of debugging, but its behavior is different in multithreaded applications. If a breakpoint is created at a source line used by multiple threads, then every thread is affected by the breakpoint. We can limit this by specifying the thread to be affected. For example:

 (gdb) break pos.c:17 thread 5 

This installs a breakpoint at line 20 in myfile.c , but only for thread number 5. We can further refine these breakpoints using thread qualifiers. For example:

  11  void *posThread(void *arg)  12  {  13  int ret;  14   15  ret = checkPosition(arg);  16   17  if (ret == 0) {  18   19  ret = move(arg);  20  }  21  (gdb) b pos.c:17 thread 5 if ret > 0           Breakpoint 1 at 0x8048550: file pos.c, line 19           (gdb) 

In this example, we specify to break at line 17 in file pos.c for thread 5, but here we qualify that thread 5 will be stopped only if the local ret variable is greater than 0.

We can identify the threads that are currently active in a multithreaded application using the info threads command. This command lists each of the active threads and its current state. For example:

 (gdb) info threads   5 Thread -161539152 (LWP 2819)  posThread (arg=0x0) at pos.c:17 ... * 1 Thread -151046720 (LWP 2808)  init at init.c:154 (gdb) 

The * before thread 1 identifies that it is the current focus of the debugger. We could switch to any thread using the thread command, which allows us to change the focus of the debugger to the specified thread.

 (gdb) thread 1  [Switching to thread 1 (Thread -161539152 (LWP 2819))]#0  posThread      17  if (ret == 0) {     (gdb) 

As we step through a multithreaded program, we ll find that the focus of the debugger can change at any step. This can be annoying, especially when the current thread is what we re interested in debugging. We can instruct GDB not to preempt the current thread by locking the scheduler. For example:

 (gdb) set scheduler-locking on 

This tells GDB not to preempt the current thread. When we want to allow other threads to preempt our current thread, we can set the mode to off:

 (gdb) set scheduler-locking off 

Finally, we can identify the current mode using the show command:

 (gdb) show scheduler-locking     Mode for locking scheduler during execution is on.     (gdb) 

One final important command for thread debugging is the ability to apply a single command to all threads within an application. The thread apply all command is used for this purpose. For example, the following command will emit a stack backtrace for every active thread:

 (gdb) thread apply all backtrace 

The thread apply command can also apply to a list of threads instead of all threads, as illustrated below:

 (gdb) thread apply 1 4 9 backtrace 

This performs a stack backtrace on threads 1, 4, and 9.

Debugging an Existing Process

We can debug an application that is currently running by attaching GDB to the process. All that we need is the process identifier for the process to debug. In this example, we ve started our application in one terminal and then started GDB in another. Once GDB has started, we issue the attach command to attach to the process. This suspends the process, allowing us to control it.

 $ gdb     GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)     ...     This GDB was configured as "i386-redhat-linux-gnu".     (gdb) attach 23558     Attaching to process 23558     Reading symbols from /home/mtj/gnulinux/ch25/testapp...done.     Reading symbols from /lib/tls/libc.so.6...done.     Loaded symbols for /lib/tls/libc.so.6     Reading symbols from /lib/ld-linux.so.2...done.     Loaded symbols for /lib/ld-linux.so.2     0x08048468 in operator (stack=0xbfffe9e0, op=1) at testapp.c:51     51        a = pop(stack); b = pop(stack);     (gdb) bt     #0  0x08048468 in operator (stack=0xbfffe9e0, op=1) at testapp.c:51     #1  0x080485cc in main () at testapp.c:93     #2  0x42015504 in __libc_start_main () from /lib/tls/libc.so.6     (gdb) 

Note  

This method is very useful for dealing with hung programs where the fault occurs only after some period of time, or for dealing with unexpected hangs in production environments.

GDB starts by loading the symbols for the process and then identifying where the process was suspended (in the operator function). We issue the bt command to list the backtrace, which tells us which particular invocation of operator we re in (in this case, an OP_SUBTRACT call). Finally, if we re done debugging, we can release the process to continue by detaching from it using the detach call:

 (gdb) detach     Detaching from program: /home/mtj/gnulinux/ch25/testapp,      process 23558     (gdb) quit     $ 

Once the detach command has finished, our process continues normally.

Postmortem Debugging

When an application aborts and dumps a resulting core dump file, GDB can be used to identify what happened . Our application has been hardened , but we ll remove a couple of asserts in the push function in order to force a core dump.

Note  

To enable GNU/Linux to generate a core dump, the command ulimit -c unlimited should be performed. Otherwise, with limits in place, core dump files will not be generated.

We execute our application to get the core dump:

 $ ./testapp     Segmentation fault (core dumped)     $ ls     core.23730  testapp  testapp.c 

Now that we have our core dump, we can use GDB to identify where things went wrong. In the following example, we specify the executable application and the core dump image to GDB. It loads the app and uses the core dump file to identify what happened at the time of failure. After all the symbols are loaded, we see that the function failure occurred at push (but we already knew that). What s most important is that we see someone called push with a stack argument of 0 (null pointer). We would have caught this with our assert function, but it was conveniently removed for the sake of demonstration.

Further down, we see that the offending call was made at testapp line 30. This happens to be a call that we added to force the creation of this core file.

 # gdb testapp core.23730     GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)     ...     Core was generated by `./testapp.     Program terminated with signal 11, Segmentation fault.     Reading symbols from /lib/tls/libc.so.6...done.     Loaded symbols for /lib/tls/libc.so.6     Reading symbols from /lib/ld-linux.so.2...done.     Loaded symbols for /lib/ld-linux.so.2     #0  0x0804839c in push (stack=0x0, elem=2) at testapp.c:30          30        stack->stack[stack->index++] = elem;     (gdb) bt     #0  0x0804839c in push (stack=0x0, elem=2) at testapp.c:30     #1  0x08048536 in main () at testapp.c:81     #2  0x42015504 in __libc_start_main () from /lib/tls/libc.so.6     (gdb) 

Although that was a quick review, it covers many of the necessary features that are needed for debugging with GDB.