Section 15.3. Debugging with Shared Libraries

15.3. Debugging with Shared Libraries

Now that you understand how to invoke a remote debug session using GDB on the host and gdbserver on the target, we turn our attention to the complexities of shared libraries and debug symbols. Unless your application is a statically linked executable (linked with the -static linker command line switch), many symbols in your application will reference code outside your application. Obvious examples include the use of standard C library routines such as fopen,printf,malloc, and memcpy. Less obvious examples might include calls to application-specific functions, such as jack_transport_locate() (a routine from the JACK low-latency audio server), which calls a library function outside the standard C libraries.

To have symbols from these routines available, you must satisfy two requirements for GDB:

• You must have debug versions of the libraries available.

• GDB must know where to find them.

If you don't have debug versions of the libraries available, you can still debug your application; you just won't have any debug information available for library routines called by your application. Often this is perfectly acceptable, unless, of course, you are developing a shared library object as part of your embedded project.

Look back at Listing 15-4, where we invoked GDB on a remote target. After GDB connected via the target remote command, GDB issued a two-line response:

Remote debugging using 192.168.1.141:2001 0x40000790 in ?? ()

This confirms that GDB connected to our target at the indicated IP address and port. GDB then reports the location of the program counter as 0x40000790. Why do we get question marks instead of a symbolic location? Because this is the Linux dynamic loader (ld-x.y.z.so), and on this particular platform, we do not have debug symbols available for this shared library. How do we know this?

Recall our introduction of the /proc file system from Chapter 9, "File Systems." One of the more useful entries was the maps enTRy (see Listing 9-16, in Chapter 9) in the per-process directory structure. We know the process ID (PID) of our target application from the gdbserver output in Listing 15-3. Our process was assigned PID 197. Given that, we can see the memory segments in use right after process startup, as shown in Listing 15-6.

Listing 15-6. Initial Target Memory Segment Mapping

root@coyote:~# cat /proc/197/maps 00008000-00026000 r-xp 00000000 00:0e 4852444    ./websdemo-stripped 0002d000-0002e000 rw-p 0001d000 00:0e 4852444    ./websdemo-stripped 40000000-40017000 r-xp 00000000 00:0a 4982583    /lib/ld-2.3.3.so 4001e000-40020000 rw-p 00016000 00:0a 4982583    /lib/ld-2.3.3.so bedf9000-bee0e000 rwxp bedf9000 00:00 0          [stack] root@coyote:~#

Here we see the target websdemo-stripped application occupying two memory segments. The first is the read-only executable segment at 0x8000, and the second is a read-write data segment at 0x2d000. The third memory segment is the one of interest. It is the Linux dynamic linker's executable code segment. Notice that it starts at address 0x40000000. If we investigate further, we can confirm that GDB is actually sitting at the first line of code for the dynamic linker, before any code from our own application has been executed. Using our cross version of readelf, we can confirm the starting address of the linker as follows:

# xscale_be-readelf -S ld-2.3.3.so | grep \.text [ 9] .text    PROGBITS    00000790 000790 012c6c 00  AX  0   0 16

From this data, we conclude that the address GDB reports on startup is the first instruction from ld-2.3.3.so, the Linux dynamic linker/loader. You can use this technique to get rough ideas of where your code is if you don't have symbolic debug information for a process or shared library.

Remember that we are executing this cross readelf command on our development host. Therefore, the ld-2.3.3.so file, itself an XScale binary object, must be accessible to your development host. Most typically, this file resides on your development host, and is a component of your embedded Linux distribution installed on your host.

15.3.1. Shared Library Events in GDB

GDB can alert you to shared library events. This can be useful for understanding your application's behavior or the behavior of the Linux loader, or for setting breakpoints in shared library routines you want to debug or step through. Listing 15-7 illustrates this technique. Normally, the complete path to the library is displayed. This listing has been edited for better readability.

Listing 15-7. Stopping GDB on Shared Library Events

$ xscale_be-gdb -q websdemo (gdb) target remote 192.168.1.141:2001 Remote debugging using 192.168.1.141:2001 0x40000790 in ?? () (gdb) i shared       <<<Display loaded shared libs No shared libraries loaded at this time. (gdb) b main         <<<Break at main Breakpoint 1 at 0x12b80: file main.c, line 72. (gdb)  c Continuing. Breakpoint 1, main (argc=0x1, argv=0xbec7fdc4) at main.c:72 72               int localvar = 9; (gdb) i shared From        To          Syms Read   Shared Object Library 0x40033300  0x4010260c  Yes         /opt/mvl/.../lib/tls/libc.so.6 0x40000790  0x400133fc  Yes         /opt/mvl/.../lib/ld-linux.so.3 (gdb) set stop-on-solib-events 1 (gdb)  c Continuing. Stopped due to shared library event (gdb)  i shared From        To          Syms Read   Shared Object Library 0x40033300  0x4010260c  Yes         /opt/mvl/.../lib/tls/libc.so.6 0x40000790  0x400133fc  Yes         /opt/mvl/.../lib/ld-linux.so.3 0x4012bad8  0x40132104  Yes         /opt/mvl/.../libnss_files.so.2 (gdb)

When the debug session is first started, of course, no shared libraries are loaded. You can see this with the first i shared command. This command displays the shared libraries that are currently loaded. Setting a breakpoint at our application's main() function, we see that two shared libraries are now loaded. These are the Linux dynamic linker/loader and the standard C library component libc.

From here, we issue the set stop-on-solib-event command and continue program execution. When the application tries to execute a function from another shared library, that library is loaded. In case you are wondering, the gethostbyname() function is encountered and causes the next shared object load.

This example illustrates an important cross-development concept. The binary application (ELF image) running on the target contains information on the libraries it needs to resolve its external references. We can view this information easily using the ldd command introduced in Chapter 11, "BusyBox," and detailed in Chapter 13. Listing 15-8 shows the output of ldd invoked from the target board.

Listing 15-8. ldd Executed on Target Board

root@coyote:/workspace# ldd websdemo          libc.so.6 => /lib/tls/libc.so.6 (0x40020000)          /lib/ld-linux.so.3    (0x40000000) root@coyote:/workspace#

Notice that the paths to the shared libraries on the target are absolute paths starting at /lib on the root file system. But GDB running on your host development workstation cannot use these paths to find the libraries. You should realize that to do so would result in your host GDB loading libraries from the wrong architecture. Your host is likely x86, whereas, in this example, the target is ARM XScale.

If you invoke your cross version of ldd, you will see the paths that were preconfigured into your toolchain. Your toolchain must have knowledge of where these files exist on your host development system.^[2] Listing 15-9 illustrates this. Again, we have edited the listing for readability; long paths have been abbreviated.

^[2] It is certainly possible to pass these locations to your compiler, linker, and debugger for every invocation, but any good embedded Linux distribution will configure these defaults into the toolchain as a convenience to the developer.

Listing 15-9. ldd Executed on Development Host

$ xscale_be-ldd    websdemo    libc.so.6 => /opt/mvl/.../xscale_be/target/lib/libc.so.6 (0xdead1000)    ld-linux.so.3 => /opt/mvl/.../xscale_be/target/lib/ld-linux.so.3 (0xdead2000) $

Your cross toolchain should be preconfigured with these library locations. Not only does your host GDB need to know where they are located, but, of course, your compiler and linker also need this knowledge.^[3] GDB can tell you where it is configured to look for these libraries using the show solib-absolute-prefix command:

^[3] Of course, your compiler also needs to know the location of target files such as architecture-specific system and library header files.

(gdb) show solib-absolute-prefix Prefix for loading absolute shared library symbol files is "/opt/mvl/pro/devkit/arm/xscale_be/target". (gdb)

You can set or change where GDB searches for shared libraries using the GDB commands set solib-absolute-prefix and set solib-search-path. If you are developing your own shared library modules or have custom library locations on your system, you can use solib-search-path to instruct GDB where to look for your libraries. For more details about these and other GDB commands, consult the online GDB manual referenced at the end of this chapter in Section 15.6.1, "Suggestions for Additional Reading."

One final note about ldd. You might have noticed the addresses from Listing 15-8 and 15-9 associated with the libraries. ldd displays the load address for the start of these code segments as they would be if the program were loaded by the Linux dynamic linker/loader. Executed on the target, the addresses in Listing 15-5 make perfect sense, and we can correlate these with the /proc/<pid>/maps listing of the running process on the target. Listing 15-10 displays the memory segments for this target process after it is completely loaded and running.

Listing 15-10. Memory Segments from /proc/<pid>/maps on Target

root@coyote:~# cat /proc/197/maps 00008000-00026000 r-xp 00000000 00:0e 4852444    /workspace/websdemo-stripped 0002d000-0002e000 rw-p 0001d000 00:0e 4852444    /workspace/websdemo-stripped 0002e000-0005e000 rwxp 0002e000 00:00 0          [heap] 40000000-40017000 r-xp 00000000 00:0a 4982583    /lib/ld-2.3.3.so 40017000-40019000 rw-p 40017000 00:00 0 4001e000-4001f000 r--p 00016000 00:0a 4982583    /lib/ld-2.3.3.so 4001f000-40020000 rw-p 00017000 00:0a 4982583    /lib/ld-2.3.3.so 40020000-4011d000 r-xp 00000000 00:0a 4982651    /lib/tls/libc-2.3.3.so 4011d000-40120000 ---p 000fd000 00:0a 4982651    /lib/tls/libc-2.3.3.so 40120000-40124000 rw-p 000f8000 00:0a 4982651    /lib/tls/libc-2.3.3.so 40124000-40126000 r--p 000fc000 00:0a 4982651    /lib/tls/libc-2.3.3.so 40126000-40128000 rw-p 000fe000 00:0a 4982651    /lib/tls/libc-2.3.3.so 40128000-4012a000 rw-p 40128000 00:00 0 4012a000-40133000 r-xp 00000000 00:0a 4982652    /lib/tls/libnss_files-2.3.3.so 40133000-4013a000 ---p 00009000 00:0a 4982652    /lib/tls/libnss_files-2.3.3.so 4013a000-4013b000 r--p 00008000 00:0a 4982652    /lib/tls/libnss_files-2.3.3.so 4013b000-4013c000 rw-p 00009000 00:0a 4982652    /lib/tls/libnss_files-2.3.3.so becaa000-becbf000 rwxp becaa000 00:00 0          [stack] root@coyote:~#

Notice the correlation of the target ldd output from Listing 15-8 to the memory segments displayed in the /proc file system for this process. The start (beginning of .text segment) of the Linux loader is 0x40000000 and the start of libc is at 0x40020000. These are the virtual addresses where these portions of the application have been loaded, and are reported by the target invocation of ldd. However, the load addresses reported by the cross version of ldd in Listing 15-9 (0xdead1000 and 0xdead2000) are there to remind you that these libraries cannot be loaded on your host system (they are ARM architecture binaries), and the load addresses are simply placeholders.