Section 19.4. Pickled Objects | Programming Python

19.4. Pickled Objects

Probably the biggest limitation of DBM keyed files is in what they can store: data stored under a key must be a simple text string. If you want to store Python objects in a DBM file, you can sometimes manually convert them to and from strings on writes and reads (e.g., with str and eval calls), but this takes you only so far. For arbitrarily complex Python objects such as class instances and nested data structures, you need something more. Class instance objects, for example, cannot be later re-created from their standard string representations. Custom to-string conversions are error prone and not general.

The Python pickle module, a standard part of the Python system, provides the conversion step needed. It converts nearly arbitrary Python in-memory objects to and from a single linear string format, suitable for storing in flat files, shipping across network sockets between trusted sources, and so on. This conversion from object to string is often called serializationarbitrary data structures in memory are mapped to a serial string form.

The string representation used for objects is also sometimes referred to as a byte stream, due to its linear format. It retains all the content and references structure of the original in-memory object. When the object is later re-created from its byte string, it will be a new in-memory object identical in structure and value to the original, though located at a different memory address. The re-created object is effectively a copy of the original.

Pickling works on almost any Python datatypenumbers, lists, dictionaries, class instances, nested structures, and moreand so is a general way to store data. Because pickles contain native Python objects, there is almost no database API to be found; the objects stored are processed with normal Python syntax when they are later retrieved.

19.4.1. Using Object Pickling

Pickling may sound complicated the first time you encounter it, but the good news is that Python hides all the complexity of object-to-string conversion. In fact, the pickle module 's interfaces are incredibly simple to use. For example, to pickle an object into a serialized string, we can either make a pickler and call its methods or use convenience functions in the module to achieve the same effect:

P = pickle.Pickler( file): Make a new pickler for pickling to an open output file object file.
P.dump( object): Write an object onto the pickler's file/stream.
pickle.dump( object, file): Same as the last two calls combined: pickle an object onto an open file.
string = pickle.dumps( object): Return the pickled representation of object as a character string.

Unpickling from a serialized string back to the original object is similarboth object and convenience function interfaces are available:

U = pickle.Unpickler( file): Make an unpickler for unpickling from an open input file object file.
object = U.load( ): Read an object from the unpickler's file/stream.
object = pickle.load( file): Same as the last two calls combined: unpickle an object from an open file.
object = pickle.loads( string): Read an object from a character string rather than a file.

Pickler and Unpickler are exported classes. In all of the preceding cases, file is either an open file object or any object that implements the same attributes as file objects:

Pickler calls the file's write method with a string argument.
Unpickler calls the file's read method with a byte count, and readline without arguments.

Any object that provides these attributes can be passed in to the file parameters. In particular, file can be an instance of a Python class that provides the read/write methods (i.e., the expected file-like interface). This lets you map pickled streams to in-memory objects with classes, for arbitrary use. For instance, the StringIO standard library module discussed in Chapter 3 provides classes that map file calls to and from in-memory strings.

This hook also lets you ship Python objects across a network, by providing sockets wrapped to look like files in pickle calls at the sender, and unpickle calls at the receiver (see the sidebar "Making Sockets Look Like Files," in Chapter 13, for more details). In fact, for some, pickling Python objects across a trusted network serves as a simpler alternative to network transport protocols such as SOAP and XML-RPC; provided that Python is on both ends of the communication (pickled objects are represented with a Python-specific format, not with XML text).

19.4.2. Picking in Action

In more typical use, to pickle an object to a flat file, we just open the file in write mode and call the dump function:

 % python >>> table = {'a': [1, 2, 3],              'b': ['spam', 'eggs'],              'c': {'name':'bob'}} >>> >>> import pickle >>> mydb  = open('dbase', 'w') >>> pickle.dump(table, mydb)

Notice the nesting in the object pickled herethe pickler handles arbitrary structures. To unpickle later in another session or program run, simply reopen the file and call load:

 % python >>> import pickle >>> mydb  = open('dbase', 'r') >>> table = pickle.load(mydb) >>> table {'b': ['spam', 'eggs'], 'a': [1, 2, 3], 'c': {'name': 'bob'}}

The object you get back from unpickling has the same value and reference structure as the original, but it is located at a different address in memory. This is true whether the object is unpickled in the same or a future process. In Python-speak, the unpickled object is == but is not is:

 % python >>> import pickle >>> f = open('temp', 'w') >>> x = ['Hello', ('pickle', 'world')]           # list with nested tuple >>> pickle.dump(x, f) >>> f.close( )                                   # close to flush changes >>> >>> f = open('temp', 'r') >>> y = pickle.load(f) >>> y ['Hello', ('pickle', 'world')] >>> >>> x == y, x is y (True, False)

To make this process simpler still, the module in Example 19-1 wraps pickling and unpickling calls in functions that also open the files where the serialized form of the object is stored.

Example 19-1. PP3E\Dbase\filepickle.py

 import pickle def saveDbase(filename, object):     file = open(filename, 'w')     pickle.dump(object, file)          # pickle to file     file.close( )                     # any file-like object will do def loadDbase(filename):     file = open(filename, 'r')     object = pickle.load(file)         # unpickle from file     file.close( )                     # re-creates object in memory     return object

To store and fetch now, simply call these module functions; here they are in action managing a fairly complex structure with multiple references to the same nested objectthe nested list called L at first is stored only once in the file:

 C:\...\PP3E\Dbase>python >>> from filepickle import * >>> L = [0] >>> D = {'x':0, 'y':L} >>> table = {'A':L, 'B':D}              # L appears twice >>> saveDbase('myfile', table)          # serialize to file C:\...\PP3E\Dbase>python >>> from filepickle import * >>> table = loadDbase('myfile')         # reload/unpickle >>> table {'B': {'x': 0, 'y': [0]}, 'A': [0]} >>> table['A'][0] = 1                   # change shared object >>> saveDbase('myfile', table)          # rewrite to the file C:\...\PP3E\Dbase>python >>> from filepickle import * >>> print loadDbase('myfile')           # both L's updated as expected {'B': {'x': 0, 'y': [1]}, 'A': [1]}

Besides built-in types like the lists, tuples, and dictionaries of the examples so far, class instances may also be pickled to file-like objects. This provides a natural way to associate behavior with stored data (class methods process instance attributes) and provides a simple migration path (class changes made in module files are automatically picked up by stored instances). Here's a brief interactive demonstration:

 >>> class Rec:         def _ _init_ _(self, hours):             self.hours = hours         def pay(self, rate=50):             return self.hours * rate >>> bob = Rec(40) >>> import pickle >>> pickle.dump(bob, open('bobrec', 'w')) >>> >>> rec = pickle.load(open('bobrec')) >>> rec.hours 40 >>> rec.pay( ) 2000

We'll explore how this works in more detail in conjunction with shelves later in this chapteras we'll see, although the pickle module can be used directly, it is also the underlying translation engine in both shelves and ZODB databases.

In fact, Python can pickle just about anything, except for:

Compiled code objects; functions and classes record just their names in pickles, to allow for later reimport and automatic acquisition of changes made in module files.
Instances of classes that do not follow class importability rules (more on this at the end of the section "Shelve Files," later in this chapter).
Instances of some built-in and user-defined types that are coded in C or depend upon transient operating system states (e.g., open file objects cannot be pickled).

A PicklingError is raised if an object cannot be pickled.

19.4.3. Pickler Protocols and cPickle

In recent Python releases, the pickler introduced the notion of protocolsstorage formats for pickled data. Specify the desired protocol by passing an extra parameter to the pickling calls (but not to unpickling calls: the protocol is automatically determined from the pickled data):

 pickle.dump(object, file, protocol)

Pickled data may be created in either text or binary protocols. By default, the storage protocol is text (also known as protocol 0). In text mode, the files used to store pickled objects may be opened in text mode as in the earlier examples, and the pickled data is printable ASCII text, which can be read (it's essentially instructions for a stack machine).

The alternative protocols (protocols 1 and 2) store the pickled data in binary format and require that files be opened in binary mode (e.g., rb, wb). Protocol 1 is the original binary format; protocol 2, added in Python 2.3, has improved support for pickling of new-style classes. Binary format is slightly more efficient, but it cannot be inspected. An older option to pickling calls, the bin argument, has been subsumed by using a pickling protocol higher than 0. The pickle module also provides a HIGHEST_PROTOCOL variable that can be passed in to automatically select the maximum value.

One note: if you use the default text protocol, make sure you open pickle files in text mode later. On some platforms, opening text data in binary mode may cause unpickling errors due to line-end formats on Windows:

 >>> f = open('temp', 'w')                  # text mode file on Windows >>> pickle.dump(('ex', 'parrot'), f)       # use default text protocol >>> f.close( )  >>> >>> pickle.load(open('temp', 'r'))         # OK in text mode ('ex', 'parrot') >>> pickle.load(open('temp', 'rb'))        # fails in binary Traceback (most recent call last):   File "<pyshell#337>", line 1, in -toplevel-     pickle.load(open('temp', 'rb'))  ...lines deleted... ValueError: insecure string pickle

One way to sidestep this potential issue is to always use binary mode for your files, even for the text pickle protocol. Since you must open files in binary mode for the binary pickler protocols anyhow (higher than the default 0), this isn't a bad habit to get into:

 >>> f = open('temp', 'wb')                 # create in binary mode >>> pickle.dump(('ex', 'parrot'), f)       # use text protocol >>> f.close( ) >>> >>> pickle.load(open('temp', 'rb')) ('ex', 'parrot') >>> pickle.load(open('temp', 'r')) ('ex', 'parrot')

Refer to Python's library manual for more information on the pickler. Also check out marshal, a module that serializes an object too, but can handle only simple object types. pickle is more general than marshal and is normally preferred.

And while you are flipping (or clicking) through that manual, be sure to also see the entries for the cPickle modulea reimplementation of pickle coded in C for faster performance. You can explicitly import cPickle for a substantial speed boost; its chief limitation is that you cannot subclass its versions of Pickle and Unpickle because they are functions, not classes (this is not required by most programs). The pickle and cPickle modules use compatible data formats, so they may be used interchangeably.

If it is available in your Python, the shelve module automatically chooses the cPickle module for faster serialization, instead of pickle. I haven't explained shelve yet, but I will now.