Recipe 19.20. Running an Iterator in Another ThreadCredit: Garth Kidd ProblemYou want to run the code of a generator (or any other iterator) in its own separate thread, so that the iterator's code won't block your main thread even if it contains time-consuming operations, such as blocking calls to the operating system. SolutionThis task is best tackled by wrapping a subclass of threading.Thread around the iterator: import sys, threading class SpawnedGenerator(threading.Thread): def _ _init_ _(self, iterable, queueSize=0): threading.Thread._ _init_ _(self) self.iterable = iterable self.queueSize = queueSize def stop(self): "Ask iteration to stop as soon as feasible" self.stopRequested = True def run(self): "Thread.start runs this code in another, new thread" put = self.queue.put try: next = iter(self.iterable).next while True: # report each result, propagate StopIteration put((False, next( )) if self.stopRequested: raise StopIteration except: # report any exception back to main thread and finish put((True, sys.exc_info( ))) def execute(self): "Yield the results that the "other", new thread is obtaining" self.queue = Queue.Queue(self.queueSize) get = self.queue.get self.stopRequested = False self.start( ) # executes self.run( ) in other thread while True: iterationDone, item = get( ) if iterationDone: break yield item # propagate any exception (unless it's just a StopIteration) exc_type, exc_value, traceback = item if not isinstance(exc_type, StopIteration): raise exc_type, exc_value, traceback def _ _iter_ _(self): "Return an iterator for our executed self" return iter(self.execute( )) DiscussionGenerators (and other iterators) are a great way to package the logic that controls an iteration and obtains the next value to feed into a loop's body. The code of a generator (and, equivalently, the code of the next method of another kind of iterator) usually runs in the same thread as the code that's iterating on it. The "calling" code can therefore block, each time around the loop, while waiting for the generator's code to do its job. Sometimes, you want to use a generator (or other kind of iterator) in a "non-blocking" way, which means you need to arrange things so that the generator's body runs in a new, separate thread. This recipe shows a class which supplies exactly this kind of functionality: this recipe's SpawnedGenerator class subclasses threading.Thread and uses Thread's start/run mechanism to ensure the generator's body always executes in a separate thread from that of the calling code. All communication between the two threads occurs through a single instance of the Queue.Queue class (held through a local-variable bound method in each of the communicating methods: the generator named execute that runs in the calling thread and the method named run that runs in a separate thread). The "calling" code may also call method stop on the SpawnedGenerator instance to ask for the iteration to stop as soon as feasible. Optionally, you may also specify a queue size when you instantiate SpawnedGenerator, if you want to limit how far ahead of the calling thread the spawned thread can get. The main use case for this recipe is for wrapping iterators that make blocking calls to the operating system (e.g., walking a directory tree), when you need to use such iterators in an application where the "main" thread cannot be allowed to block for a long time. The typical examples of applications whose main thread must not block are event-driven applications, a description that applies to applications with a GUI, as well as to networking applications built on asynchronous frameworks, such as Twisted or the asyncore module of the Python Standard Library. See AlsoLibrary Reference and Python in a Nutshell docs about modules threading and asyncore; Twisted is at http://www.twistedmatrix.com/; Chapter 9 for general issues about threading; Chapter 11 for general issues about user interfaces; Chapter 13 and Chapter 14 for general issues about network and web programming, including asynchronous approaches to such programs. |