Recipe17.10.Debugging Memory Problems


Recipe 17.10. Debugging Memory Problems

Credit: Will Ware

Problem

You're developing C extensions, and you experience memory problems. You suspect mismanagement of reference counts and want to check whether your C extension code is correctly managing reference counts.

Solution

To chase these problems in an optimal way, you need to alter Python's sources and rebuild Python. Specifically, add the following function in Objects/object.c, immediately before the _Py_PrintReferences function:

void _Py_CountReferences(FILE *fp) {     int nr, no;     PyObject *op;     for (nr = no = 0, op = refchain._ob_next;          op != &refchain;          op = op->_ob_next, nr += op->ob_refcnt, no += 1)     { }     fprintf(fp, "%d refs (%d), %d objs\n", nr, _Py_RefTotal, no); }

I place the following macros in my C extensions:

#if defined(Py_DEBUG) || defined(DEBUG) extern void _Py_CountReferences(FILE*); #define CURIOUS(x) { fprintf(stderr, _ _FILE_ _ ":%d ", _ _LINE_ _); x; } #else #define CURIOUS(x) #endif #define MARKER( )        CURIOUS(fprintf(stderr, "\n")) #define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d\n", x)) #define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x\n", x)) #define COUNTREFS( )     CURIOUS(_Py_CountReferences(stderr))

To debug, I rebuild Python using make OPT="-DPy_DEBUG", which causes the code under Py_TRACE_REFS to be built. My own makefile for my extensions uses the same trick by including these lines:

debug:         make clean; make OPT="-g -DPy_DEBUG" all CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python2.3

Discussion

When I'm developing C extensions and running into memory problems, I find that the typical cause is mismanagement of reference counts, particularly misuse of Py_INCREF and Py_DECREF, as well as forgetfulness of the reference-count effects of functions like Py_BuildValue, PyArg_ParseTuple, and PyTuple/List_SetItem/GetItem. The Python sources offer help with this problem (search for Py_TRACE_REFS), and function sys.getrefcounts in the Python Standard Library is also helpful. Nevertheless, it's useful to add this recipe's function in Objects/object.c just before _Py_PrintReferences.

Unlike _Py_PrintReferences, this recipe's _Py_CountReferences function prints only the totals of all the refcounts and number of objects in the system, so it can be sensibly called, even in loops that repeat millions of times, while _Py_PrintReferences would print out way too much stuff to be useful. The information printed by _Py_CountReferences can help you identify errantly wandering Py_INCREFs and Py_DECREFs. _Py_CountReferences plays it safe by performing its own counts of objects references, which it prints side by side with the "official" count of references that Python itself maintains (when compiled for debugging) as global variable _Py_RefTotal. Should any discrepancy arise, you know something deeply wrong is going on.

When I suspect that one of my C-coded functions is responsible for memory problems, I liberally sprinkle the suspect function with calls to the COUNTREFS macro. Doing so allows me to keep track of exactly how many references are being created or destroyed as I go through my function. This information is particularly useful in tight loops, in which dumb mistakes can cause reference counts to grow ridiculously fast. Also, reference counts that shrink too fast (because of overzealous use of Py_DECREF) can cause core dumps because the memory for objects that should still exist has been reallocated for new objects.

See Also

The only documentation in this case is Python's own source code. Use the source, Luke!



Python Cookbook
Python Cookbook
ISBN: 0596007973
EAN: 2147483647
Year: 2004
Pages: 420

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net