Section 15.4. Debugging Multiple Tasks

15.4. Debugging Multiple Tasks

Generally the developer is presented with two different debugging scenarios when dealing with multiple threads of execution. Processes can exist in their own address space or can share an address space (and other system resources) with other threads of execution. The former (independent processes not sharing common address space) must be debugged using separate independent debug sessions. Nothing prevents you from using gdbserver on multiple processes on your target system, and using a separate invocation of GDB on your development host to coordinate a debug session for multiple cooperating but independent processes.

15.4.1. Debugging Multiple Processes

When a process being debugged under GDB uses the fork() system call ^[4] to spawn a new process, GDB can take two courses of action. It can continue to control and debug the parent process, or it can stop debugging the parent process and attach to the newly formed child process. You can control this behavior using the set follow-fork-mode command. The two modes are follow parent and follow child. The default behavior is for GDB to follow the parent. In this case, the child process executes immediately upon a successful fork.

^[4] We will use the term system call, but fork() in this context is actually the C library function which in turn calls the Linux sys_fork() system call.

Listing 15-11 reproduces a snippet of a simple program that forks multiple processes from its main() routine.

Listing 15-11. Using fork() to Spawn a Child Process

...   for( i=0; i<MAX_PROCESSES; i++ ) {     /* Creating child process */     pid[i] = fork();                /* Parent gets non-zero PID */     if ( pid[i] == -1 ) {       perror("fork failed");       exit(1);     }     if ( pid[i] == 0 ) {      /* Indicates child's code path */        worker_process();      /* The forked process calls this */     }   }   /* Parent's main control loop */   while ( 1 ) { ...   }

This simple loop creates MAX_THREADS new processes using the fork() system call. Each newly spawned process executes a body of code defined by the function worker_process(). When run under GDB in the default mode, GDB detects the creation of the new threads of execution (processes) but remains attached to the parent's thread of execution. Listing 15-12 illustrates this GDB session.

Listing 15-12. GDB in follow-fork-mode = parent

(gdb) target remote 192.168.1.141:2001 0x40000790 in ?? () (gdb)  b main Breakpoint 1 at 0x8888: file forker.c, line 104. (gdb)  c Continuing. [New Thread 356] [Switching to Thread 356] Breakpoint 1, main (argc=0x1, argv=0xbe807dd4) at forker.c:104 104       time(&start_time); (gdb)  b worker_process Breakpoint 2 at 0x8784: file forker.c, line 45. (gdb)  c Continuing. Detaching after fork from child process 357. Detaching after fork from child process 358. Detaching after fork from child process 359. Detaching after fork from child process 360. Detaching after fork from child process 361. Detaching after fork from child process 362. Detaching after fork from child process 363. Detaching after fork from child process 364.

Notice that eight child processes were spawned, with PID values from 357 to 364. The parent process was instantiated with PID 356. When the breakpoint in main() was hit, we entered a breakpoint at the worker_process() routine, which each child process executes upon fork(). Letting the program continue from main, we see each of the new processes spawned and detached by the debugger. They never hit the breakpoint because GDB is attached to the main process, which never executes the worker_process() routine.

If you need to debug each process, you must execute a separate independent GDB session and attach to the child process after it is forked(). The GDB documentation referenced at the end of this chapter outlines a useful technique to place a call to sleep() in the child process, giving you time to attach a debugger to the new process. Attaching to a new process is explained in Section 15.5.2, "Attaching to a Running Process."

If you simply need to follow the child process, set the follow-fork-mode to follow child before your parent reaches the fork() system call. Listing 15-13 shows this.

Listing 15-13. GDB in follow-fork-mode = child

(gdb) target remote 192.168.1.141:2001 0x40000790 in ?? () (gdb) set follow-fork-mode child (gdb)  b worker_process Breakpoint 1 at 0x8784: file forker.c, line 45. (gdb)  c Continuing. [New Thread 401] Attaching after fork to child process 402. [New Thread 402] [Switching to Thread 402] Breakpoint 1, worker_process () at forker.c:45 45         int my_pid = getpid(); (gdb)  c Continuing.

Here we see the parent process being instantiated as PID 401. When the first child is spawned by the fork() system call, GDB detaches silently from the parent thread of execution and attaches to the newly spawned child process having PID 402. GDB is now in control of the first child process and honors the breakpoint set at worker_process(). Notice, however, that the other child processes spawned by the code snippet from Listing 15-11 are not debugged and continue to run to their own completion.

In summary, using GDB in this fashion, you are limited to debugging a single process at a time. You can debug through the fork() system call, but you have to decide which thread of execution to follow through the fork() call, either the parent or the child. As mentioned in the introduction to this section, you can use multiple independent GDB sessions if you must debug more than one cooperating process at a time.

15.4.2. Debugging Multithreaded Applications

If your application uses the POSIX thread library for its threading functions, GDB has additional capabilities to handle concurrent debugging of a multithreaded application. The Native Posix Thread Library (NPTL) has become the de facto standard thread library in use on Linux systems, including embedded Linux systems. The rest of this discussion assumes that you are using this thread library.

For this section, we use a demonstration program that spawns a number of threads using the pthread_create() library function in a simple loop. After the threads are spawned, the main() routine simply waits for keyboard input to terminate the application. Each thread displays a short message on the screen and sleeps for a predetermined time. Listing 15-14 shows the startup sequence on the target board.

Listing 15-14. Target Threads Demo Startup

root@coyote:/workspace # gdbserver localhost:2001 ./tdemo Process ./tdemo created; pid = 671 Listening on port 2001 Remote debugging from host 192.168.1.10     ^^^^^  Previous three lines displayed by gdbserver tdemo main() entered: My pid is 671 Starting worker thread 0 Starting worker thread 1 Starting worker thread 2 Starting worker thread 3

As in our previous examples, gdbserver prepares the application for running and waits for a connection from our host-based cross-gdb. When GDB connects, gdbserver reports the connection with the Remote debugging... message. Now we start GDB on the host and connect. Listing 15-15 reproduces this half of the session.

Listing 15-15. Host GDB Connecting to Target Threads Demo

$ xscale_be-gdb -q tdemo (gdb) target remote 192.168.1.141:2001 0x40000790 in ?? () (gdb)   b tdemo.c:97 Breakpoint 1 at 0x88ec: file tdemo.c, line 97. (gdb)   c Continuing. [New Thread 1059] [New Thread 1060] [New Thread 1061] [New Thread 1062] [New Thread 1063] [Switching to Thread 1059] Breakpoint 1, main (argc=0x1, argv=0xbefffdd4) at tdemo.c:98 98               int c = getchar(); (gdb)

Here we connect to the target (resulting in the "Remote debugging..." message in Listing 15-14), set a breakpoint just past the loop where we spawned the new threads, and continue. When the new thread is created, GDB displays a notice along with the thread ID. Thread 1059 is the TDemo application, doing its work directly from the main() function. Threads 1060 through 1063 are the new threads created from the call to pthread_create().

When GDB hits the breakpoint, it displays the message [Switching to Thread 1059], indicating that this was the thread of execution that encountered the breakpoint. It is the active thread for the debugging session, referred to as the current thread in the GDB documentation.

GDB enables us to switch between threads and perform the usual debugging operations such as setting additional breakpoints, examining data, displaying a backtrace, and working with the individual stack frames within the current thread. Listing 15-16 provides examples of these operations, continuing directly with our debugging session started in Listing 15-15.

Listing 15-16. GDB Operations on Threads

... (gdb) c Continuing.                   <<< Ctl-C to interrupt program execution Program received signal SIGINT, Interrupt. 0x400db9c0 in read () from /opt/mvl/.../lib/tls/libc.so.6 (gdb)  i threads   5 Thread 1063  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6   4 Thread 1062  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6   3 Thread 1061  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6   2 Thread 1060  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6 * 1 Thread 1059  0x400db9c0 in read ()    from /opt/mvl/.../lib/tls/libc.so.6 (gdb) thread 4               <<< Make Thread 4 the current thread [Switching to thread 4 (Thread 1062)] #0  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6 (gdb)  bt #0  0x400bc714 in nanosleep ()    from /opt/mvl/.../lib/tls/libc.so.6 #1  0x400bc4a4 in __sleep (seconds=0x0) at sleep.c:137 #2  0x00008678 in go_to_sleep (duration=0x5) at tdemo.c:18 #3  0x00008710 in worker_2_job (random=0x5) at tdemo.c:36 #4  0x00008814 in worker_thread (threadargs=0x2) at tdemo.c:67 #5  0x40025244 in start_thread (arg=0xfffffdfc) at pthread_create.c:261 #6  0x400e8fa0 in clone () at../sysdeps/unix/sysv/linux/arm/clone.S:82 #7  0x400e8fa0 in clone () at../sysdeps/unix/sysv/linux/arm/clone.S:82 (gdb)   frame 3 #3  0x00008710 in worker_2_job (random=0x5) at tdemo.c:36 36          go_to_sleep(random); (gdb)  l                    <<< Generate listing of where we are 31      } 32 33      static void worker_2_job(int random) 34      { 35          printf("t2 sleeping for %d\n", random); 36          go_to_sleep(random); 37      } 38 39      static void worker_3_job(int random) 40      { (gdb)

A few points are worth mentioning. GDB assigns its own integer value to each thread and uses these values to reference the individual threads. When a breakpoint is hit in a thread, all threads within the process are halted for examination. GDB marks the current thread with an asterisk (*). You can set unique breakpoints within each threadassuming, of course, that they exist in a unique context. If you set a breakpoint in a common portion of code where all threads execute, the thread that hits the breakpoint first is arbitrary.

The GDB user documentation referenced at the end of this chapter contains more useful information related to debugging in a multithreaded environment.

15.4.3. Debugging Bootloader/Flash Code

Debugging Flash resident code presents its own unique challenges. The most obvious limitation is the way in which GDB and gdbserver cooperate in setting target breakpoints. When we discussed the GDB remote serial protocol in Chapter 14, you learned how breakpoints are inserted into an application.^[5] GDB replaces the opcode at the breakpoint location with an architecture-specific opcode that passes control to the debugger. However, in ROM or Flash, GDB cannot overwrite the opcode, so this method of setting breakpoints is useless.

^[5] Refer back to Listing 14-5 in Chapter 14.

Most modern processors contain some number of debug registers that can be used to get around this limitation. These capabilities must be supported by architecture-and processor-specific hardware probes or stubs. The most common technique for debugging Flash and ROM resident code is to use JTAG hardware probes. These probes support the setting of processor-specific hardware breakpoints. This topic was covered in detail in Chapter 14. Refer back to Section 14.4.2, "Debugging with a JTAG Probe," for details.