Recipe8.4.Trapping and Recording Exceptions

Recipe 8.4. Trapping and Recording Exceptions

Credit: Mike Foord

Problem

You need to trap exceptions, record their tracebacks and error messages, and then proceed with the execution of your program.

Solution

A typical case is a program that processes many independent files one after the other. Some files may be malformed and cause exceptions. You need to trap such exceptions, record the error information, then move on to process subsequent files. For example:

import cStringIO, traceback def process_all_files(all_filenames,                       fatal_exceptions=(KeyboardInterrupt, MemoryError)                      ):     bad_filenames = {  }     for one_filename in all_filenames:         try:             process_one_file(one_filename):         except fatal_exceptions:             raise         except Exception:             f = cStringIO.StringIO( )             traceback.print_exc(file=f)             bad_filenames[one_filename] = f.getvalue( )     return bad_filenames

Discussion

Because Python exceptions are very powerful tools, you need a clear and simple strategy to deal with them. This recipe will probably not fit your needs exactly, but it may be a good starting point from which to develop the right strategy for your applications.

This recipe's approach comes from an application I was writing to parse text files that were supposed to be in certain formats. Malformed files could easily cause exceptions, and I needed to get those errors' tracebacks and messages to either fix my code to be more forgiving or fix malformed files; however, I also needed program execution to continue on subsequent files.

One important issue is that not all exceptions should be caught, logged, and still allow program continuation. A KeyboardInterrupt exception means the user is banging on Ctrl-C (or Ctrl-Break, or some other key combination), specifically asking for your application to stop; we should, of course, honor such requests, not ignore them. A MemoryError means you have run out of memoryunless you've got huge caches of previous results that you can immediately delete to make more memory available, generally you can't do much about such a situation. Depending on your application and exact circumstances, other errors might well also be deemed just as fatal. So, process_all_files accepts a fatal_exceptions argument, a tuple of exception classes it should not catch (but which it should rather propagate), defaulting to the pair of exception classes we just mentioned. The try/except statement is carefully structured to catch, and re-raise, any exception in those classes, with precedence over the general except Exception handler clause, which catches everything else.

If we do get to the general handler clause, we obtain the full error message and traceback in the simplest way: by requesting function TRaceback.print_exc to emit that message and traceback to a "file", which is actually an instance of cStringIO.StringIO, a "file"-like object specifically designed to ease in-memory capture of information from functions that normally write to files. The getvalue method of the StringIO instance provides the message and traceback as a string, and we store the string in dictionary bad_filenames, using, as the corresponding key, the filename that appears to have caused the probl7em. process_all_files' for loop then moves on to the next file it must process.

Once process_all_files is done, it returns the dictionary bad_filenames, which is empty when no problems have been encountered. Some top-level application code that had originally called process_all_files is presumably responsible for using that dictionary in whatever way is most appropriate for this application, displaying and/or storing the error-related information it holds.

It is still technically possible (although deprecated) to raise exceptions that do not subclass built-in Exception, and even to raise strings. If you need to catch such totally anomalous cases (whose possibility will no doubt stay around for years for backwards compatibility), you need to add one last unconditional except clause to your try/except statement:

        except fatal_exceptions:             raise         except Exception:             ...         except:             ...

Of course, if what you want to do for all normal (nonfatal) exceptions, and for the weird anomalous cases, is exactly the same, you don't need a separate except Exception clausejust the unconditional except clause will do. However, you may normally want to log the occurrence of the weird anomalous cases in some different and more prominent way, because, these days (well into the twenty-first century), they're definitely not expected under any circumstance whatsoever.

Recipe8.4.Trapping and Recording Exceptions