Recipe8.3.Debugging the Garbage-Collection Process

Recipe 8.3. Debugging the Garbage-Collection Process

Credit: Dirk Holtwick

Problem

You know that memory is leaking from your program, but you have no indication of what exactly is being leaked. You need more information to help you figure out where the leaks are coming from, so you can remove them and lighten the garbage-collection work periodically performed by the standard gc module.

Solution

The gc module lets you dig into garbage-collection issues:

import gc def dump_garbage( ):     """ show us what the garbage is about """     # Force collection     print "\nGARBAGE:"     gc.collect( )     print "\nGARBAGE OBJECTS:"     for x in gc.garbage:         s = str(x)         if len(s) > 80: s = s[:77]+'...'         print type(x),"\n  ", s if _ _name_ _=="_ _main_ _":     gc.enable( )     gc.set_debug(gc.DEBUG_LEAK)     # Simulate a leak (a list referring to itself) and show it     l = [  ]     l.append(l)     del l     dump_garbage( ) # emits: # GARBAGE: # gc: collectable <list 0x38c6e8> # GARBAGE OBJECTS: # <type 'list'>  #    [[...]]

Discussion

In addition to the normal debugging output of gc, this recipe shows the garbage objects, to help you get an idea of where the leak may be. Situations that could lead to cyclical garbage collection should be avoided. Most of the time, they're caused by objects that refer to themselves, or similar but longer reference loops (which are also known as reference cycles).

Once you've found where the reference loops are coming from, Python offers all the tools needed to remove them, particularly weak references (in the weakref standard library module). But especially in big programs, you first have to get an idea of where to find the leak before you can remove it and enhance your program's performance. For this purpose, it's good to know what the objects being leaked contain, and the dump_garbage function in this recipe comes in quite handy on such occasions.

This recipe works by first calling gc.set_debug to tell the gc module to keep the leaked objects in its gc.garbage list rather than recycling them. Then, this recipe's dump_garbage function calls gc.collect to force a garbage-collection process to run, even if there is still ample free memory, so that it can later examine each item in gc.garbage and print out its type and contents (limiting the printout to no more than 80 characters per garbage object, to avoid flooding the screen with huge chunks of information).

See Also

Documentation for the gc and weakref modules in the Library Reference and Python in a Nutshell.