Section 7.1. Using the GNU Debugger

7.1. Using the GNU Debugger

If software applications were bug-free, we would not have any use for debuggers. As software programmers, we all strive to write good programs. As the code gets bigger and more complicated, however, simple programming mistakes can creep in. Like software development, porting an application can introduce bugs in the program. Even if the application is ported without changing any application programming interfaces (APIs) used in the code, a hidden bug in the original code can suddenly expose itself simply by running the application on a different platform. This is one reason why you need to be familiar with the debugger available on Linux.

The GNU debugger, GDB, is a powerful debugger for different applications written in C, C++, Objective-C, FORTRAN, and Ada. This section shows how to use the GDB for multiprocess or multithreaded applications. It serves as a quick how-to guide for anyone new to GDB. The information contained in this section deals with the basics of the GNU debugger; for more definitive information regarding the use of GDB, consult the GNU documentation project for GDB on the Internet.^[1]

^[1] www.gnu.org/software/gdb/documentation

7.1.1. Compiling a Program for Use with GDB

To get the most effective use out of GDB, the application to be debugged (the debugee) needs to be compiled with the g flag, as follows:

$ gcc g file.c o file

When compiling using makefiles, the g flag may be included in CFLAGS or CPPFLAGS, as in the following example:

DEBUG_FLAG = -g CFLAGS = $(DEBUG_FLAGS) D_USING_APA

%.o: %.c $(CC) -c $(CFLAGS) $< -o $@

The gcc compiler supports the g flag with O (optimization), making it possible to debug optimized executable programs with GDB. The one thing to remember, though, is that debugging an optimized program may cause some execution path to be rearranged and not match the source files.

7.1.2. Invoking GDB

There are several ways to invoke GDB, as follows:

Invoking GDB by specifying the executable program as its argument.
```
$ gdb sample_programs/main_program 
```
(Optional) Passing arguments to the executable program.
```
$ gdb sample_programs/main_program a x 
```
Invoking GDB with both an executable program and a core file as its arguments.
```
$ gdb sample_programs/main_program core_file 
```
Invoking GDB with an executable program name and a process ID.
```
$ gdb sample_programs/main_program 1234 
```
In this example, GDB attaches^[2] to process 1234, which is assumed to be the process ID of the executable, main_program.
^[2] Use the command detach from the gdb prompt to resume execution of the running program.
Invoking GDB with command-line options affects how GDB starts up. An example of invoking GDB without printing the startup message by using the silent flag follows:
```
$ gdb silent sample_programs/main_program core_file 
```

7.1.3. GDB Basic Commands

For the most part, commands used inside GDB are intuitive. Once inside GDB, commands to show back traces or source line numbers can easily be remembered. Table 7-1 is a list of the basic GDB commands most widely used when debugging a ported application on Linux.

Table 7-1. List of Basic GDB Commands
GDB Commands	Action
`list, show`	List source line number
`break, clear`	Set and delete breakpoints
`bt, where`	List back trace of current thread of execution
`step, next`	Advance program to the next line of execution
`continue`	Continue executing the program
`stepi, nexti`	Execute one machine instruction
`finish`	Continue running until just after the selected stack frame returns
`info`	Return information on various parts of the program
`print`	Examine data
`X`	Examine memory
`directory, dir`	Source directories to use for examining source files

For commands that are not frequently used, the porting engineer can get help using the info command.

7.1.4. Debugging Multiple Processes

On most Linux platforms, GDB has no special support to debug programs that create child processes using the fork function. When a program forks, GDB continues to debug the parent process while the child continues to execute in the background, as shown in Example 7-1.

Example 7-1. Code Listing for fork_sample.c

1 2 #include <sys/types.h> 3 #include <unistd.h> 4 #include <sys/wait.h> 5 6 main() 7 { 8   pid_t child; 9   pid_t child_pid; 10  int status; 11 12   if ((child = fork()) < 0) 13    { 14     printf("fork failed\n"); 15     exit(1); 16    } 17   else if (child == 0) 18    { 19     /* child */ 20     child_pid = getpid(); 21      printf("I am the child %d\n", child_pid); 22    } 23   else 24    { 25     /* parent */ 26      printf("I am the parent %d\n", getpid()); 27     waitpid(child_pid, &status, WNOHANG); 28    } 29 30   exit(0); 31 }

Compile fork_sample.c:

$ gcc fork_sample. c o fork_sample g    compile code with -g

Run GDB against the compiled code:

$ gdb fork_sample (gdb) break printf              put a breakpoint in printf Breakpoint 1 at 0xfecadf0 (gdb) run Starting program: fork_sample Detaching after fork from child process 8012. I am the child 8012 Breakpoint 1, 0x0fecadf0 in printf () from /lib/tls/libc.so.6 (gdb) where #0 0x0fecadf0 in printf () from /lib/tls/libc.so.6 #1 0x10000548 in main () at fork.c:26

In the sample GDB session, we placed a breakpoint at a printf statement. But how do we know which printf statement the debugger will stop atthe child's or the parent's? The session shows GDB stopped at the printf invocation inside the parent process (line 26). GDB does not have any special provisions to create a separate GDB session to debug the child process. To follow the child process using GDB, use the command set follow-fork-mode within GDB. The following GDB session shows how to follow a child using GDB:

$ gdb fork_sample (gdb) set follow-fork-mode child    Set GDB to follow the child (gdb) break printf Breakpoint 1 at 0x10010aa8 (gdb) run Starting program: fork_sample Breakpoint 1 at 0xfecadf0 Attaching after fork to child process 8019. I am the parent 8016 [Switching to process 8019] Breakpoint 1, 0x0fecadf0 in printf () from /lib/tls/libc.so.6 (gdb) where #0 0x0fecadf0 in printf () from /lib/tls/libc.so.6 #1 0x10000528 in main () at fork.c:21

Notice that after setting follow-fork-mode to follow the child, GDB stopped at line 21, where the printf function was about to be invoked by the child process.

7.1.5. Debugging Multithreaded Applications

You need to remember only a few thread-specific commands when using GDB to debug multithreaded applications. The rest of the GDB commands to get stack traces and other program information remain the same. The thread-specific commands for use in GDB are as follows:

info threadsA command to inquire about existing threads
thread <thread number>A command to switch threads
thread apply [thread number] [all] argsA command to apply a command to a list of threads

The program and debugging session shown in Example 7-2 is an example of how to debug a multithreaded application with GDB.

Example 7-2. Listing of thread_sample.c

1 2 #include <pthread.h> 3 #include <signal.h> 4 #include <errno.h> 5 #include <sched.h> 6 #include <stdio.h> 7 #include <stdlib.h> 8 #include <unistd.h> 9 10 #define THREAD_NUM 3 11 12 13 int 14 common_func(int x) 15 { 16   sleep(x); 17 } 18 19 void * 20 func1(void *arg) 21 { 22   common_func(5); 23   printf("thread func1 exiting\n"); 24 } 25 26 void * 27 func2(void *arg) 28 { 29    common_func(10); 30    printf("thread func2 exiting\n"); 31 } 32 33 void * 34 func3(void *arg) 35 { 36    common_func(15); 37    printf("thread func3 exiting\n"); 38 } 39 40 void *(*func_array[THREAD_NUM])(void *) = { 41             func1, 42             func2, 43             func3 44             }; 45 46 int 47 main(void) 48 { 49   pthread_t tid[THREAD_NUM]; 50   int i = 0; 51   int rc = 0; 52   void *(*fp[THREAD_NUM])(void *); 53 54    for (i=0; i<THREAD_NUM; i++) 55    { 56     fp[i] = func_array[i]; 57    } 58 59    for (i=0; i<THREAD_NUM; i++) 60    { 61     if( (rc = pthread_create(&tid[i],NULL, fp[i],NULL))!= 0) 62     { 63       printf("pthread_create failed: %d\n", errno); 64       exit (1); 65     } 66    } 67 68    for (i=0; i<THREAD_NUM; i++) 69    { 70     pthread_join(tid[i], NULL); 71    } 72 73    return (0); 74 } 75

Compile the sample multithreaded application:

$ gcc thread_sample.c -o thread_sample -g -lpthread -D_REENTRANT

Run GDB against the sample program:

$ gdb thread_sample (gdb) run Starting program: thread_sample [Thread debugging using libthread_db enabled] [New Thread 4160589856 (LWP 8051)] [New Thread 4160586976 (LWP 8054)] [New Thread 4150101216 (LWP 8055)] [New Thread 4139615456 (LWP 8056)] ctl-c               ctl-c to break into the debugger Program received signal SIGINT, Interrupt. [Switching to Thread 4160589856 (LWP 8051)] 0x0fc268a8 in pthread_join () from /lib/tls/libpthread.so.0 (gdb) info thread  4 Thread 4139615456 (LWP 8056) 0x0ff15a64 in __nanosleep_nocancel ()   from /lib/tls/libc.so.6  3 Thread 4150101216 (LWP 8055) 0x0ff15a64 in __nanosleep_nocancel ()   from /lib/tls/libc.so.6  2 Thread 4160586976 (LWP 8054) 0x0ff15a64 in __nanosleep_nocancel ()   from /lib/tls/libc.so.6 * 1 Thread 4160589856 (LWP 8051) 0x0fc268a8 in pthread_join ()   from /lib/tls/libpthread.so.0

Notice the output of the info thread command: One of the lines has an asterisk. The asterisk signifies this thread is the current thread about which GDB will display information:

(gdb) where              display stack trace #0 0x0fc268a8 in pthread_join () from /lib/tls/libpthread.so.0 #1 0x1000079c in main () at thread_sample.c:70

To make GDB switch to a different thread, issue the command tHRead <num> where num is the thread number taken from the info thread output:

(gdb) thread 3 [Switching to thread 3 (Thread 4150101216 (LWP 8059))]#0 0x0ff15a64 in __nanosleep_nocancel () from /lib/tls/libc.so.6 (gdb) where #0 0x0ff15a64 in __nanosleep_nocancel () from /lib/tls/libc.so.6 #1 0x0ff157e8 in sleep () from /lib/tls/libc.so.6 #2 0x10000564 in common_func (x=10) at thread_sample.c:16 #3 0x100005e8 in func2 (arg=0x0) at thread_sample.c:29 #4 0x0fc25a58 in start_thread () from /lib/tls/libpthread.so.0 #5 0x0ff51c5c in clone () from /lib/tls/libc.so.6 #6 0x0ff51c5c in clone () from /lib/tls/libc.so.6

To cause a GDB command to apply to all threads, issue the tHRead apply all command. The following example applies the where (display back trace) command to all threads:

(gdb) thread apply all where Thread 4 (Thread 4139615456 (LWP 8060)): #0 0x0ff15a64 in __nanosleep_nocancel () from /lib/tls/libc.so.6 #1 0x0ff157e8 in sleep () from /lib/tls/libc.so.6 #2 0x10000564 in common_func (x=15) at thread_sample.c:16 #3 0x10000630 in func3 (arg=0x0) at thread_sample.c:36 #4 0x0fc25a58 in start_thread () from /lib/tls/libpthread.so.0 #5 0x0ff51c5c in clone () from /lib/tls/libc.so.6 #6 0x0ff51c5c in clone () from /lib/tls/libc.so.6 Previous frame identical to this frame (corrupt stack?) Thread 3 (Thread 4150101216 (LWP 8059)): #0 0x0ff15a64 in __nanosleep_nocancel () from /lib/tls/libc.so.6 #1 0x0ff157e8 in sleep () from /lib/tls/libc.so.6 #2 0x10000564 in common_func (x=10) at thread_sample.c:16 #3 0x100005e8 in func2 (arg=0x0) at thread_sample.c:29 #4 0x0fc25a58 in start_thread () from /lib/tls/libpthread.so.0 #5 0x0ff51c5c in clone () from /lib/tls/libc.so.6 #6 0x0ff51c5c in clone () from /lib/tls/libc.so.6 Previous frame identical to this frame (corrupt stack?) Thread 2 (Thread 4160586976 (LWP 8058)): #0 0x0ff15a64 in __nanosleep_nocancel () from /lib/tls/libc.so.6 #1 0x0ff157e8 in sleep () from /lib/tls/libc.so.6 #2 0x10000564 in common_func (x=5) at thread_sample.c:16 #3 0x100005a0 in func1 (arg=0x0) at thread_sample.c:22 #4 0x0fc25a58 in start_thread () from /lib/tls/libpthread.so.0 #5 0x0ff51c5c in clone () from /lib/tls/libc.so.6 #6 0x0ff51c5c in clone () from /lib/tls/libc.so.6 Previous frame identical to this frame (corrupt stack?) Thread 1 (Thread 4160589856 (LWP 8057)): #0 0x0fc268a8 in pthread_join () from /lib/tls/libpthread.so.0 #1 0x1000079c in main () at thread_sample.c:70

7.1.6. Stopping Multithreaded Programs

If the application being debugged is a multithreaded program, as in the preceding example, it is possible to set breakpoints on a particular thread by using the following GDB commands:

(gdb) break <line> thread <thread number>

(gdb) break <line> thread <thread number> if <qualifier>

where line is a particular line number in the specified module used in the program and tHRead number is the thread number taken from the info thread output. The following is a sample invocation of the command:

(gdb) break thread_sample.c:14 thread 2

To conditionally stop a thread, use a conditional qualifier as follows:

(gdb) break thread_sample.c:14 thread 2 if foo < bar

Remember that whenever GDB stops a particular thread, all threads in the program are stopped, too. Threads that are in the middle of a system call are interrupted and may return from the system call without completing the system call. It is therefore suggested that programs check system call return values so that the proper action can be taken if the call is interrupted.

When the multithreaded program is restarted, all threads are started, too. When doing a step or next command, GDB does not guarantee that all threads will be stopped after executing a line or instruction in the program. Some threads may execute more than a single line or instruction depending on where the threads are in their current state of execution.

7.1.7. GDB Initialization File

During startup, GDB reads an initialization file if it exists. This file is read from the user's home directory and/or the current working directory and is normally named .gdbinit. The .gdbinit file is useful for customizing a GDB session. GDB reads commands from the .gdbinit file and executes them for the particular GDB session.

An example of where .gdbinit can prove useful is when specifying source directories so that GDB can find source file information. Another example is setting the child follow-fork-mode before starting GDB, as in the following example:

$ cat .gdbinit dir /home/user1/src/lib:/home/user1/src/unix set follow-fork-mode child

7.1.8. GDB and Signals

Like any normal application executing outside of GDB, applications under the control of GDB (debuggee) may receive signals from external sources. GDB can control the signals directed at the debuggee by either passing them to the debuggee or ignoring them. The commands to control signal behavior under GDB are as follows:

info signals or info handle Prints a table of all the kinds of signals and how GDB has been told to handle each one. You can use this command to see the signal numbers of all the defined types of signals.
handle signal keywords . . . Changes the way GDB handles a specific signal. The signal can be the number of a signal or its name (with or without the SIG at the beginning); a list of signal numbers of the form low-high; or the word all, meaning all the known signals. The keywords say what changes to make.
Keywords:
nostop GDB should not stop your program when this signal happens. It may still print a message telling you that the signal has come in.
stop GDB should stop your program when this signal happens. This implies the print keyword, too.
print GDB should print a message when this signal happens.
noprint GDB should not mention the occurrence of the signal at all. This implies the nostop keyword, too.
pass or noignore GDB should allow your program to see this signal; your program can handle the signal, or else it may terminate if the signal is fatal.

Example 7-3 shows a GDB session using a sample program that invokes some signals.

Example 7-3. Listing of signal_sample.c

1 #include <stdio.h> 2 #include <signal.h> 3 #include <errno.h> 4 5 void sig_handler(int signo) 6 { 7    printf("in alarm_handler()\n"); 8    alarm(5); /* reset alarm */ 9 } 10 11 int main() 12 { 13   sigset_t sigmask; 14   struct sigaction sigact; 15 16   sigact.sa_handler=sig_handler; 17   sigact.sa_flags=0; 18 19   if (sigaction(SIGALRM, &sigact, NULL) < 0) 20   { 21    perror("sigaction()"); 22    return 1; 23   } 24 25    alarm(6); 26 27    while (1) 28   { 29     sleep(1); 30     fprintf(stdout, "."); 31     fflush(stdout); 32   } 33 }

Compile the sample program with the g flag:

$ gcc signal_sample.c -o signal_sample g

Invoke GDB on the compiled program:

$ gdb signal_sample (gdb) run Starting program: /home/avm/book/GDB/signal_sample ..... in alarm_handler() ..... in alarm_handler() ..... in alarm_handler() .....in alarm_handler() ctl-c (gdb) info signal Signal    Stop    Print  Pass to program  Description SIGHUP    Yes    Yes     Yes             Hangup SIGINT    Yes    Yes     No              Interrupt SIGQUIT   Yes    Yes     Yes             Quit SIGILL    Yes    Yes     Yes             Illegal instruction SIGTRAP   Yes    Yes     No              Trace/breakpoint trap SIGABRT   Yes    Yes     Yes             Aborted SIGEMT    Yes    Yes     Yes             Emulation trap SIGFPE    Yes    Yes     Yes             Arithmetic exception SIGKILL   Yes    Yes     Yes             Killed SIGBUS    Yes    Yes     Yes             Bus error SIGSEGV   Yes    Yes     Yes             Segmentation fault SIGSYS    Yes    Yes     Yes             Bad system call SIGPIPE   Yes    Yes     Yes             Broken pipe SIGALRM   No     No      Yes             Alarm clock SIGTERM   Yes    Yes     Yes             Terminated SIGURG    No     No      Yes             Urgent I/O condition SIGSTOP   Yes    Yes     Yes             Stopped (signal) ...

Notice in the preceding output that SIGALRM has the Print column set to No. If that is changed to a Yes through the handle command, GDB exhibits the following behavior:

(gdb) handle SIGALRM print             change SIGALRM Print status to Yes Signal      Stop   Print  Pass to program  Description SIGALRM     No     Yes    Yes              Alarm clock (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/avm/book/GDB/signal_sample ..... Program received signal SIGALRM, Alarm clock. in alarm_handler() ..... Program received signal SIGALRM, Alarm clock. in alarm_handler() ..... Program received signal SIGALRM, Alarm clock. in alarm_handler() ...

Notice that after changing the SIGALRM print state to Yes, GDB now prints a status message every time the program receives a SIGALRM. In the following example, we change the stop state of SIGALRM to Yes, which causes GDB to stop when the program receives a SIGALRM:

(gdb) handle SIGALRM stop             change SIGALRM Stop status to Yes Signal       Stop   Print  Pass to program   Description SIGALRM      Yes    Yes    Yes               Alarm clock (gdb) run                             run the application The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/avm/book/GDB/signal_sample ..... Program received signal SIGALRM, Alarm clock. 0x0ff15a30 in __nanosleep_nocancel () from /lib/tls/libc.so.6 (gdb)

After the SIGALRM stop state was changed to Yes and the program restarted, GDB now stops every time a SIGALRM is generated.

7.1.9. Graphical Front End to GDB

For developers who are more used to graphical debuggers, there is a graphical front end available for GDB called DDD.^[3] DDD is available from www.gnu.org/software/ddd/.

^[3] DDD can also be used as a front end for debuggers other than GDB.

For developers who prefer an Integrated Development Environment (IDE), both open-source and commercial products are available. Eclipse^[4] and Kdevelop^[5] are available as open-source IDEs. Visual Slickedit^[6] and Borland Kylix^[7] are commercial IDEs available for Linux.

^[4] www.eclipse.org

^[5] www.kdevelop.org

^[6] www.slickedit.com

^[7] www.borland.com