Recipe19.19.Simplifying Queue-Consumer Threads

Recipe 19.19. Simplifying Queue-Consumer Threads

Credit: Jimmy Retzlaff, Paul Moore

Problem

You want to code a consumer thread which gets work requests off a queue one at a time, processes each work request, and eventually stops, and you want to code it in the simplest possible way.

Solution

This task is an excellent use case for the good old Sentinel idiom. The producer thread, when it's done putting actual work requests on the queue, must finally put a sentinel value, that is, a value that is different from any possible work request. Schematically, the producer thread will do something like:

for input_item in stuff_needing_work:     work_request = make_work_request(input_item)     queue.put(work_request) queue.put(sentinel)

where sentinel must be a "well-known value", different from any work_request object that might be put on the queue in the first phase.

The consumer thread can then exploit the built-in function iter:

for work_request in iter(queue.get, sentinel):     process_work_request(work_request) cleanup_and_terminate( )

Discussion

Were it not for built-in function iter, the consumer thread would have to use a slightly less simple and elegant structure, such as:

while True:     work_request = queue.get( )     if work_request == sentinel:         break     process_work_request(work_request) cleanup_and_terminate( )

However, the Sentinel idiom is so useful and important that Python directly supports it with built-in function iter. When you call iter with just one argument, that argument must be an iterable object, and iter returns an iterator for it. But when you call iter with two arguments, the first one must be a callable which can be called without arguments, and the second one is an arbitrary value known as the sentinel. In the two-argument case, iter repeatedly calls the first argument. As long as each call returns a value !=sentinel, that value becomes an item in the iteration; as soon as a call returns a value ==sentinel, the iteration stops.

If you had to code this yourself as a generator, you could write:

def iter_sentinel(a_callable, the_sentinel):     while True:         item = a_callable( )         if item == the_sentinel: break         yield item

But the point of this recipe is that you don't have to code even this simple generator: just use the power that Python gives you as part of the functionality of the built-in function iter!

Incidentally, Python offers many ways to make sentinel valuesmeaning values that compare equal only to themselves. The simplest and most direct way, and therefore the one I suggest you always use for this specific purpose, is:

sentinel = object( )

Recipe19.19.Simplifying Queue-Consumer Threads