Recipe16.10.Referring to a List Comprehension While Building It

Recipe 16.10. Referring to a List Comprehension While Building It

Credit: Chris Perkins

Problem

You want to refer, from inside a list comprehension, to the same list object you're building. However, the object being built by the list comprehension doesn't have a name while you're building it.

Solution

Internally, the Python interpreter does create a "secret" name that exists only while a list comprehension is being built. In Python 2.3, that name is usually '_[1]' and refers to the bound method append of the list object we're building. We can use this secret name as a back door to refer to the list object as it gets built. For example, say we want to build a copy of a list but without duplicates:

>>> L = [1, 2, 2, 3, 3, 3] >>> [x for x in L if x not in locals( )['_[1]']._ _self_ _] [1, 2, 3]

Python 2.4 uses the same name to indicate the list object being built, rather than the bound-method access. In the case of nested list comprehensions, inner ones are named '_[2]', '_[3]', and so on, as nesting goes deeper. Clearly, all of these considerations are best wrapped up into a function:

import inspect import sys version_23 = sys.version_info < (2, 4) def this_list( ):     import sys     d = inspect.currentframe(1).f_locals     nestlevel = 1     while '_[%d]' % nestlevel in d: nestlevel += 1     result = d['_[%d]' % (nestlevel - 1)]     if version_23: return result._ _self_ _     else: return result

Using this function, we can make the preceding snippet more readable, as well as making it work properly in Python 2.4 as well as in version 2.3:

>>> [x for x in L if x not in this_list( )] [1, 2, 3]

Discussion

List comprehensions may look a little like magic, but the bytecode that Python generates for them is in fact quite mundane: create an empty list, give the empty list's bound-method append, a temporary name in the locals dictionary, append items one at a time, and then delete the name. All of this happens, conceptually, between the open square bracket ( [ ) and the close square bracket ( ]), which enclose the list comprehension.

The temporary name that Python 2.3 assigns to the bound append method is '_[1]' (or '_[2]', etc., for nested list comprehensions). This name is deliberately chosen (to avoid accidental clashes) to not be a syntactically valid Python identifier, so we cannot refer to the bound method directly, by name. However, we can access it as locals( )['_[1]']. Once we have a reference to the bound method object, we just use the bound method's _ _self_ _ attribute to get at the list object itself. In Python 2.4, the same name refers directly to the list object, rather than to its bound method, so we skip the last step.

Having a reference to the list object enables us to do all sorts of neat party tricks, such as performing if tests that involve looking at the items that have already been added to the list, or even modifying or deleting them. These capabilities are just what the doctor ordered for finding primes in a "one-liner", for example: for each odd number, we need to test whether it is divisible by any prime number less than or equal to the square root of the number being tested. Since we already have all the smaller primes stored and, with our new parlor trick, have access to them, this test is a breeze and requires no auxiliary storage:

import itertools def primes_less_than(N):     return [ p for p in itertools.chain([2], xrange(3,N,2))              if 0 not in itertools.imap(                  lambda x: p % x, itertools.takewhile(                        lambda v: v*v <= p, this_list( ) ))]

The list comprehension that's the whole body of this function primes_less_than, while long enough not to fit into a single physical line, is all in a single logical line (indeed, it must be, since any list comprehension is a single expression), and therefore qualifies as a "one-liner" if you squint in just the right way.

This simple prime-finding algorithm is nowhere near as fast as the Sieve of Eratosthenes shown in Recipe 18.10, but the ability to fit the entire algorithm inside a single expression is nevertheless kind of neat. Part of its neatness comes from the just-in-time evaluation that the functions from standard library module itertools perform so nicely.

Alas, this neat trick definitely cannot be recommended for production code. While it works in Python 2.3 and 2.4, it could easily break in future releases, since it depends on undocumented internals; for the same reason, it's unlikely to work properly on other implementations of the Python language, such as Jython or IronPython. So, I suggest you use it to impress friends, but for any real work, stick to clearer, faster, and solid good old for loops!

Recipe16.10.Referring to a List Comprehension While Building It