Recipe17.9.Debugging Dynamically Loaded C Extensions with gdb

Recipe 17.9. Debugging Dynamically Loaded C Extensions with gdb

Credit: Joe VanAndel, Michael Aivazis

Problem

A dynamically loaded C/C++ Python extension is giving you trouble on a Unix or Unix-like platform, and you want to use the interactive debugger gdb to determine what's wrong.

Solution

One way to determine the cause of core dumps or other serious trouble with a C Python extension is to compile the extension source with -g and then follow these steps. (You may also want to recompile any other extensions you use, such as Numeric, with -g, if you hadn't built them that way in the first place.)

<m>% gdb /usr/bin/python2.1</m> (gdb) br _PyImport_LoadDynamicModule (gdb) run    # start python (gdb) cont   # repeat until your extension is loaded (gdb) # you may need an import statement at python's >>> prompt (gdb) finish # finish loading your extension module (gdb) br wrap_myfunction  # break at the entry point in your code (gdb) disable 1   # don't break for any more modules being loaded (gdb) cont   # back to Python, run things normally from here

Discussion

If a dynamically loaded C/C++ extension is causing Python to core dump, or causing some other kind of serious trouble, this recipe can help you determine the root cause, by demonstrating a technique for debugging your extension using gdb (if you use Unix or some Unix-like platform, and gdb is your debugger of choice). The overall concept generalizes to other debuggers with abilities similar to gdb's.

The main point of this recipe is that you cannot set a break on your function at the start, because your function lives in a dynamic library (shared object) that isn't initially loaded. However, you can break in the PyImport_LoadDynamicModule function, and eventually (when your module is finally being loaded) get control at the debugger prompt right after your module is in memory. You are then able, at last, to set the breakpoints you need.

This technique works. However, if you do this kind of thing often, the process of stepping through all the modules, as Python loads them at startup, can easily become tedious. A handier alternative, although more invasive, requires you to modify your Python sources and rebuild Python from them.

The key idea of this handier alternative is to add a do-nothing function somewhere in the body of code that Python loads immediately. Specifically, you can edit the Modules/main.c file, adding one new function:

void Py_DebugTrap(void) { }

In the extension you're debugging, you can now add a call to Py_DebugTrap( ) right where you want to break into the code. The Py_DebugTrap( ) symbol is immediately available when you start gdb, because the symbol lives in main.c. So you can immediately set a breakpoint there, as soon as you are at the gdb prompt, then continue. This approach even works in parallel under MPI (message passing interface).

Recipe17.9.Debugging Dynamically Loaded C Extensions with gdb