Recipe 9.8. Multitasking Cooperatively Without Threads
Credit: Brian Bush, Troy Melhase, David Beach, Martin Miller
You have a task that seems suited to multithreading, but you don't want to incur the overhead that real thread-switching would impose.
Generators were designed to simplify iteration, but they're also quite suitable as a basis for cooperative multitasking, also known as microthreading :
import signal # credit: original idea was based on an article by David Mertz # http://gnosis.cx/publish/programming/charming_python_b7.txt # some example 'microthread' generators def empty(name): """ This is an empty task for demonstration purposes. """ while True: print "<empty process>", name yield None def terminating(name, maxn): """ This is a counting task for demonstration purposes. """ for i in xrange(maxn): print "Here %s, %s out of %s" % (name, i, maxn) yield None print "Done with %s, bailing out after %s times" % (name, maxn) def delay(duration=0.8): """ Do nothing at all for 'duration' seconds. """ import time while True: print "<sleep %d>" % duration time.sleep(duration) yield None class GenericScheduler(object): def _ _init_ _(self, threads, stop_asap=False): signal.signal(signal.SIGINT, self.shutdownHandler) self.shutdownRequest = False self.threads = threads self.stop_asap = stop_asap def shutdownHandler(self, n, frame): """ Initiate a request to shutdown cleanly on SIGINT.""" print "Request to shut down." self.shutdownRequest = True def schedule(self): def noop( ): while True: yield None n = len(self.threads) while True: for i, thread in enumerate(self.threads): try: thread.next( ) except StopIteration: if self.stop_asap: return n -= 1 if n==0: return self.threads[i] = noop( ) if self.shutdownRequest: return if _ _name_ _== "_ _main_ _": s = GenericScheduler([ empty('boo'), delay( ), empty('foo'), terminating('fie', 5), delay(0.5), ], stop_asap=True) s.schedule( ) s = GenericScheduler([ empty('boo'), delay( ), empty('foo'), terminating('fie', 5), delay(0.5), ], stop_asap=False) s.schedule( )
Microthreading (or cooperative multitasking) is an important technique. If you want to
A simple approach to cooperative multitasking, such as the one presented in this recipe, is
suitable when your tasks must perform long-running work, particularly I/O
David Mertz's site, chock-full of idiosyncratic, fascinating ideas, is at http://gnosis.cx/; Christian Tismer's Stackless Python , the best way to do cooperative multitasking in Python (and much else besides), is at http://www.stackless.com/; Twisted Matrix, the best way to do event-driven (asynchronous) programming, is at http://twistedmatrix.com/.