Recipe9.1.Synchronizing All Methods in an Object

Recipe 9.1. Synchronizing All Methods in an Object

Credit: André Bjärb, Alex Martelli, Radovan Chytracek

Problem

You want to share an object among multiple threads, but, to avoid conflicts, you need to ensure that only one thread at a time is inside the objectpossibly excepting some methods for which you want to hand-tune locking behavior.

Solution

Java offers such synchronization as a built-in feature, while in Python you have to program it explicitly by wrapping the object and its methods. Wrapping is so general and useful that it deserves to be factored out into general tools:

def wrap_callable(any_callable, before, after):     ''' wrap any callable with before/after calls '''     def _wrapped(*a, **kw):         before( )         try:             return any_callable(*a, **kw)         finally:             after( )     # In 2.4, only: _wrapped._ _name_ _ = any_callable._ _name_ _     return _wrapped import inspect class GenericWrapper(object):     ''' wrap all of an object's methods with before/after calls '''     def _ _init_ _(self, obj, before, after, ignore=( )):         # we must set into _ _dict_ _ directly to bypass _ _setattr_ _; so,         # we need to reproduce the name-mangling for double-underscores         clasname = 'GenericWrapper'         self._ _dict_ _['_%s_ _methods' % clasname] = {  }         self._ _dict_ _['_%s_ _obj' % clasname] = obj         for name, method in inspect.getmembers(obj, inspect.ismethod):             if name not in ignore and method not in ignore:                 self._ _methods[name] = wrap_callable(method, before, after)     def _ _getattr_ _(self, name):         try:             return self._ _methods[name]         except KeyError:             return getattr(self._ _obj, name)     def _ _setattr_ _(self, name, value):         setattr(self._ _obj, name, value)

Using these simple but general tools, synchronization becomes easy:

class SynchronizedObject(GenericWrapper):     ''' wrap an object and all of its methods with synchronization '''     def _ _init_ _(self, obj, ignore=( ), lock=None):         if lock is None:             import threading             lock = threading.RLock( )         GenericWrapper._ _init_ _(self, obj, lock.acquire, lock.release, ignore)

Discussion

As per usual Python practice, we can complete this module with a small self-test, executed only when the module is run as main script. This snippet also serves to show how the module's functionality can be used:

if _ _name_ _ == '_ _main_ _':     import threading     import time     class Dummy(object):         def foo(self):             print 'hello from foo'             time.sleep(1)         def bar(self):             print 'hello from bar'         def baaz(self):             print 'hello from baaz'     tw = SynchronizedObject(Dummy( ), ignore=['baaz'])     threading.Thread(target=tw.foo).start( )     time.sleep(0.1)     threading.Thread(target=tw.bar).start( )     time.sleep(0.1)     threading.Thread(target=tw.baaz).start( )

Thanks to the synchronization, the call to bar runs only when the call to foo has completed. However, because of the ignore= keyword argument, the call to baaz bypasses synchronization and thus completes earlier. So the output is:

hello from foo hello from baaz hello from bar

When you find yourself using the same single-lock locking code in almost every method of an object, use this recipe to refactor the locking away from the object's application-specific logic. The key effect you get by applying this recipe is to effectively replace each method with:

self.lock.acquire( ) try:   # The "real" application code for the method finally:     self.lock.release( )

This code idiom is, of course, the right way to express locking: the try/finally statement ensures that the lock gets released in any circumstance, whether the application code terminates correctly or raises an exception. You'll note that factory wrap_callable returns a closure, which is carefully coded in exactly this way!

To some extent, this recipe can also be handy when you want to postpone worrying about a class' locking behavior. However, if you intend to use this code for production purposes, you should understand all of it. In particular, this recipe does not wrap direct accesses (for getting or setting) to the object's attributes. If you want such direct accesses to respect the object's lock, you need to add the try/finally locking idiom to the wrapper's _ _getattr_ _ and _ _setattr_ _ special methods, around the calls these methods make to the getattr and setattr built-in functions, respectively. I normally don't find that depth of wrapping to be necessary in my applications. (The way I code, wrapping just the methods proves sufficient.)

If you're into custom metaclasses, you may be surprised that I do not offer a metaclass for these synchronization purposes. However, wrapping is a more dynamic and flexible approachfor example, an object can exist in both wrapped (synchronized) and unwrapped (raw) incarnations, and you can use the most appropriate one case by case. You pay for wrapping's flexibility with a little bit more runtime overhead at each method call, but compared to the large costs of acquiring and releasing locks I don't think this tiny extra overhead matters. Meanwhile, this recipe shows off, and effectively reuses, a wrapper-closure factory and a wrapper class that demonstrate how easy Python makes it to implement that favorite design pattern of Aspect-Oriented Programming's fans, the insertion of "before-and-after" calls around every call to an object's methods.

Recipe9.1.Synchronizing All Methods in an Object