Recipe20.15.Upgrading Class Instances Automatically on reload

Recipe 20.15. Upgrading Class Instances Automatically on reload

Credit: Michael Hudson, Peter Cogolo

Problem

You are developing a Python module that defines a class, and you're trying things out in the interactive interpreter. Each time you reload the module, you have to ensure that existing instances are updated to instances of the new, rather than the old class.

Solution

First, we define a custom metaclass, which ensures its classes keep track of all their existing instances:

import weakref class MetaInstanceTracker(type):     ''' a metaclass which ensures its classes keep track of their instances '''     def _ _init_ _(cls, name, bases, ns):         super(MetaInstanceTracker, cls)._ _init_ _(name, bases, ns)         # new class cls starts with no instances         cls._ _instance_refs_ _ = [  ]     def _ _instances_ _(cls):         ''' return all instances of cls which are still alive '''         # get ref and obj for refs that are still alive         instances = [(r, r( )) for r in cls._ _instance_refs_ _ if r( ) is not None]         # record the still-alive references back into the class         cls._ _instance_refs_ _ = [r for (r, o) in instances]         # return the instances which are still alive         return [o for (r, o) in instances]     def _ _call_ _(cls, *args, **kw):         ''' generate an instance, and record it (with a weak reference) '''         instance = super(MetaInstanceTracker, cls)._ _call_ _(*args, **kw)         # record a ref to the instance before returning the instance         cls._ _instance_refs_ _.append(weakref.ref(instance))         return instance class InstanceTracker:     ''' any class may subclass this one, to keep track of its instances '''     _ _metaclass_ _ = MetaInstanceTracker

Now, we can subclass MetaInstanceTracker to obtain another custom metaclass, which, on top of the instance-tracking functionality, implements the auto-upgrading functionality required by this recipe's Problem:

import inspect class MetaAutoReloader(MetaInstanceTracker):     ''' a metaclass which, when one of its classes is re-built, updates all         instances and subclasses of the previous version to the new one '''     def _ _init_ _(cls, name, bases, ns):         # the new class may optionally define an _ _update_ _ method         updater = ns.pop('_ _update_ _', None)         super(MetaInstanceTracker, cls)._ _init_ _(name, bases, ns)         # inspect locals & globals in the stackframe of our caller         f = inspect.currentframe( ).f_back         for d in (f.f_locals, f.f_globals):             if name in d:                 # found the name as a variable is it the old class                 old_class = d[name]                 if not isinstance(old_class, mcl):                     # no, keep trying                     continue                 # found the old class: update its existing instances                 for instance in old_class._ _instances_ _( ):                     instance._ _class_ _ = cls                     if updater: updater(instance)                     cls._ _instance_refs_ _.append(weakref.ref(instance))                 # also update the old class's subclasses                 for subclass in old_class._ _subclasses_ _( ):                     bases = list(subclass._ _bases_ _)                     bases[bases.index(old_class)] = cls                     subclass._ _bases_ _ = tuple(bases)                 break         return cls class AutoReloader:     ''' any class may subclass this one, to get automatic updates '''     _ _metaclass_ _ = MetaAutoReloader

Here is a usage example:

# an 'old class' class Bar(AutoReloader):     def _ _init_ _(self, what=23):        self.old_attribute = what # a subclass of the old class class Baz(Bar):     pass # instances of the old class & of its subclass b = Bar( ) b2 = Baz( ) # we rebuild the class (normally via 'reload', but, here, in-line!): class Bar(AutoReloader):     def _ _init_ _(self, what=42):        self.new_attribute = what+100     def _ _update_ _(self):        # compute new attribute from old ones, then delete old ones        self.new_attribute = self.old_attribute+100        del self.old_attribute     def meth(self, arg):        # add a new method which wasn't in the old class        print arg, self.new_attribute if _ _name_ _ == '_ _main_ _':     # now b is "upgraded" to the new Bar class, so we can call 'meth':     b.meth(1)     # emits: 1 123     # subclass Baz is also upgraded, both for existing instances...:     b2.meth(2)     # emits: 2 123     # ...and for new ones:     Baz( ).meth(3)     # emits: 3 142

Discussion

You're probably familiar with the problem this recipe is meant to address. The scenario is that you're editing a Python module with your favorite text editor. Let's say at some point, your module mod.py looks like this:

class Foo(object):     def meth1(self, arg):         print arg

In another window, you have an interactive interpreter running to test your code:

>>> import mod >>> f = mod.Foo( ) >>> f.meth1(1) 1

and it seems to be working. Now you edit mod.py to add another method:

class Foo(object):     def meth1(self, arg):         print arg     def meth2(self, arg):         print -arg

Head back to the test session:

>>> reload(mod) module 'mod' from 'mod.pyc' >>> f.meth2(2) Traceback (most recent call last):   File "<stdin>", line 1, in ? AttributeError: 'Foo' object has no attribute 'meth2'

Argh! You forgot that f was an instance of the old mod.Foo!

You can do two things about this situation. After reloading, either regenerate the instance:

>>> f = mod.Foo( ) >>> f.meth2(2) -2

or manually assign to f._ _class_ _:

>>> f._ _class_ _ = mod.Foo >>> f.meth2(2) -2

Regenerating works well in simple situations but can become very tedious. Assigning to the class can be automated, which is what this recipe is all about.

Class MetaInstanceTracker is a metaclass that tracks instances of its instances. As metaclasses go, it isn't too complicated. New classes of this metatype get an extra _ _instance_refs_ _ class variable (which is used to store weak references to instances) and an _ _instances_ _ class method (which strips out dead references from the _ _instance_refs_ _ list and returns real references to the still live instances). Each time a class whose metatype is MetaInstanceTracker gets instantiated, a weak reference to the instance is appended to the class' _ _instance_refs_ _ list.

When the definition of a class of metatype MetaAutoReloader executes, the namespace of the definition is examined to determine whether a class of the same name already exists. If it does, then it is assumed that this is a class redefinition, instead of a class definition, and all instances of the old class are updated to the new class. (MetaAutoReloader inherits from MetaInstanceTracker, so such instances can easily be found). All direct subclasses, found through the old class' intrinsic _ _subclasses_ _ class method, then get their _ _bases_ _ tuples rebuilt with the same change.

The new class definition can optionally include a method _ _update_ _, whose job is to update the state (meaning the set of attributes) of each instance, as the instance's class transitions from the old version of the class to the new one. The usage example in this recipe's Solution presents a case in which one attribute has changed name and is computed by different rules, as you can tell by observing the way the _ _init_ _ methods of the old and new versions are coded; in this case, the job of _ _update_ _ is to compute the new attribute based on the value of the old one, then del the old attribute for tidiness.

This recipe's code should probably do more thorough error checking; Net of error-checking issues, this recipe can also supply some fundamental tools to start solving a problem that is substantially harder than the one explained in this recipe's Problem statement: automatically upgrade classes in a long-running application, without needing to stop and restart that application.

Doing automatic upgrading in production code is more difficult than doing it during development because many more issues must be monitored. For example, you may need a form of locking to ensure the application is in a quiescent state while a number of classes get upgraded, since you probably don't want to have the application answering requests in the middle of the upgrading procedure, with some classes or instances already upgraded and others still in their old versions. You also often encounter issues of persistent storage because the application probably needs to update whatever persistent storage it keeps from old to new versions when it upgrades classes. And those are just two examples. Nevertheless, the key component of such on-the-fly upgrading, which has to do with updating instances and subclasses of old classes to new ones, can be tackled with the tools shown in this recipe.

See Also

Docs for the built-in function reload in the Library Reference and Python in a Nutshell.