Section 11.8. Variable Scope

11.8. Variable Scope

The scope of an identifier is defined to be the portion of the program where its declaration applies, or what we refer to as "variable visibility." In other words, it is like asking yourself in which parts of a program do you have access to a specific identifier. Variables either have local or global scope.

11.8.1. Global versus Local Variables

Variables defined within a function have local scope, and those at the highest level in a module have global scope. In their famous "dragon" book on compiler theory, Aho, Sethi, and Ullman summarize it this way:

"The portion of the program to which a declaration applies is called the scope of that declaration. An occurrence of a name in a procedure is said to be local to the procedure if it is in the scope of a declaration within the procedure; otherwise, the occurrence is said to be nonlocal."

One characteristic of global variables is that unless deleted, they have a lifespan that lasts as long as the script that is running and whose values are accessible to all functions, whereas local variables, like the stack frame they reside in, live temporarily, only as long as the functions they are defined in are currently active. When a function call is made, its local variables come into scope as they are declared. At that time, a new local name is created for that object, and once that function has completed and the frame deallocated, that variable will go out of scope.

global_str = 'foo' def foo():     local_str = 'bar'     return global_str + local_str

In the above example, global_str is a global variable while local_str is a local variable. The foo() function has access to both global and local variables while the main block of code has access only to global variables.

Core Note: Searching for identifiers (aka variables, names, etc.)

When searching for an identifier, Python searches the local scope first. If the name is not found within the local scope, then an identifier must be found in the global scope or else a NameError exception is raised.

A variable's scope is related to the namespace in which it resides. We will cover namespaces formally in Chapter 12; suffice it to say for now that namespaces are just naming domains that map names to objects, a virtual set of what variable names are currently in use, if you will. The concept of scope relates to the namespace search order that is used to find a variable. All names in the local namespace are within the local scope when a function is executing. That is the first namespace searched when looking for a variable. If it is not found there, then perhaps a globally scoped variable with that name can be found. These variables are stored (and searched) in the global and built-in namespaces.

It is possible to "hide" or override a global variable just by creating a local one. Recall that the local namespace is searched first, being in its local scope. If the name is found, the search does not continue to search for a globally scoped variable, hence overriding any matching name in either the global or built-in namespaces.

Also, be careful when using local variables with the same names as global variables. If you use such names in a function (to access the global value) before you assign the local value, you will get an exception (NameError or UnboundLocalError), depending on which version of Python you are using.

11.8.2. global Statement

Global variable names can be overridden by local variables if they are declared within the function. Here is another example, similar to the first, but the global and local nature of the variable are not as clear.

def foo():     print "\ncalling foo()..."     bar = 200     print "in foo(), bar is", bar bar = 100 print "in __main__, bar is", bar foo() print "\nin __main__, bar is (still)", bar

It gave the following output:

in __main__, bar is 100 calling foo()... in foo(), bar is 200 in __main__, bar is (still) 100

Our local bar pushed the global bar out of the local scope. To specifically reference a named global variable, one must use the global statement. The syntax for global is:

global var1[, var2[, ... varN]]]

Modifying the example above, we can update our code so that we use the global version of is_this_global rather than create a new local variable.

>>> is_this_global = 'xyz' >>> def foo(): ...     global is_this_global ...     this_is_local = 'abc' ...     is_this_global = 'def' ...     print this_is_local + is_this_global ... >>> foo() abcdef >>> print is_this_global def

11.8.3. Number of Scopes

Python syntactically supports multiple levels of functional nesting, and as of Python 2.1, matching statically nested scoping. However, in versions prior to 2.1, a maximum of two scopes was imposed: a function's local scope and the global scope. Even though more levels of functional nesting exist, you could not access more than two scopes:

def foo():     m = 3     def bar():         n = 4         print m + n     print m     bar()

Although this code executes perfectly fine today ...

>>> foo() 3 7

... executing it resulted in errors in Python before 2.1:

>>> foo() Traceback (innermost last):  File "<stdin>", line 1, in ?  File "<stdin>", line 7, in foo  File "<stdin>", line 5, in bar NameError: m

The access to foo()'s local variable m within function bar() is illegal because m is declared local to foo(). The only scopes accessible from bar() are bar()'s local scope and the global scope. foo()'s local scope is not included in that short list of two. Note that the output for the "print m" statement succeeded, and it is the function call to bar() that fails. Fortunately with Python's current nested scoping rules, this is not a problem today.

11.8.4. Closures

With Python's statically nested scoping, it becomes useful to define inner functions as we have seen earlier. In the next section, we will focus on scope and lambda, but inner functions also suffered the same problem before Python 2.1 when the scoping rules changed to what they are today.

If references are made from inside an inner function to an object defined in any outer scope (but not in the global scope), the inner function then is known as a closure. The variables defined in the outer function but used or referred to by the inner function are called free variables. Closures are an important concept in functional programming languages, with Scheme and Haskell being two of them. Closures are syntactically simple (as simple as inner functions) yet still very powerful.

A closure combines an inner function's own code and scope along with the scope of an outer function. Closure lexical variables do not belong to the global namespace scope or the local onethey belong to someone else's namespace and carry an "on the road" kind of scope. (Note that they differ from objects in that those variables live in an object's namespace while closure variables live in a function's namespace and scope.) So why would you want to use closures?

Closures are useful for setting up calculations, hiding state, letting you move around function objects and scope at will. Closures come in handy in GUI or event-driven programming where a lot of APIs support callbacks. The same applies for retrieving database rows and processing the data in the exact same manner. Callbacks are just functions. Closures are functions, too, but they carry some additional scope with them. They are just functions with an extra feature ... another scope.

You will probably feel that the use of closures draws a strong parallel to partial function application as introduced earlier in this chapter, but PFA is really more like currying than the use of closures because it is not as much as about function calling as it is about using variables defined in another scope.

Simple Closure Example

Below is a short example of using closures. We will simulate a counter and also simulate making an integer mutable by enclosing it as a single element of a list.

def counter(start_at=0):      count = [start_at]      def incr():          count[0] += 1          return count[0]      return incr

The only thing counter() does is to accept an initial value to start counting at and assigns it as the sole member of the list count. Then an incr() inner function is defined. By using the variable count inside it, we have created a closure because it now carries with it the scope of counter(). incr() increments the running count and returns it. Then the final magic is that counter() returns incr, a (callable) function object.

If we run this interactively, we get the output belownote how similar it looks to instantiating a counter object and executing the instance:

>>> count = counter(5) >>> print count() 6 >>> print count() 7 >>> count2 = counter(100) >>> print count2() 101 >>> print count() 8

The one difference is that we were able to do something that previously required us to write a class, and not only that, but to have to override the __call__() special method of that class to make its instances callable. Here we were able to do it with a pair of functions.

Now, in many cases, a class is the right thing to use. Closures are more appropriate in cases whenever you need a callback that has to have its own scope, especially if it is something small and simple, and often, clever. As usual, if you use a closure, it is a good idea to comment your code and/or use doc strings to explain what you are doing.

*Chasing Down Closure Lexical Variables

The next two sections contain material for advanced readers ... feel free to skip it if you wish. We will discuss how you can track down free variables with a function's func_closure attribute. Here is a code snippet that demonstrates it.

If we run this piece of code, we get the following output:

no f1 closure vars f2 closure vars: ['<cell at 0x5ee30: int object at       0x200377c>'] f3 closure vars: ['<cell at 0x5ee90: int object at       0x2003770>', '<cell at 0x5ee30: int object at       0x200377c>'] <int 'w' id=0x2003788 val=1> <int 'x' id=0x200377c val=2> <int 'y' id=0x2003770 val=3> <int 'z' id=0x2003764 val=4>

Example 11.7. Tracking Closure Variables (closureVars.py)

This example shows how we can track closure variables by using a function's func_closure variable.

1  #!/usr/bin/env python 2 3  output = '<int %r id=%#0x val=%d>' 4  w = x = y = z = 1 5 6  def f1(): 7    x = y = z = 2 8 9     def f2(): 10        y = z = 3 11 12         def f3(): 13             z = 4 14             print output % ('w', id(w), w) 15             print output % ('x', id(x), x) 16             print output % ('y', id(y), y) 17             print output % ('z', id(z), z) 18 19     clo = f3.func_closure 20     if clo: 21         print "f3 closure vars:", [str(c) for c in clo] 22     else: 23         print "no f3 closure vars" 24     f3() 25 26   clo = f2.func_closure 27   if clo: 28      print "f2 closure vars:", [str(c) for c in clo] 29   else: 30      print "no f2 closure vars" 31      f2() 32 33  clo = f1.func_closure 34  if clo: 35     print "f1 closure vars:", [str(c) for c in clo] 36  else: 37     print "no f1 closure vars" 38  f1()

Line-by-Line Explanation

Lines 14

This script starts by creating a template to output a variable: its name, ID, and value, and then sets global variables w, x, y, and z. We define the template so that we do not have to copy the same output format string multiple times.

Lines 69, 2631

The definition of the f1() function includes a creating local variables x, y, and z plus the definition of an inner function f2(). (Note that all local variables shadow or hide accessing their equivalently named global variables.) If f2() uses any variables that are defined in f1()'s scope, i.e., not global and not local to f2(), those represent free variables, and they will be tracked by f1.func_closure.

Lines 910, 1924

Practically duplicating the code for f1(), these lines do the same for f2(), which defines locals y and z plus an inner function f3(). Again, note that the locals here shadow globals as well as those in intermediate localized scopes, e.g., f1()'s. If there are any free variables for f3(), they will be displayed here.

You will no doubt notice that references to free variables are stored in cell objects, or simply, cells. What are these guys? Cells are basically a way to keep references to free variables alive after their defining scope(s) have completed (and are no longer on the stack).

For example, let us assume that function f3() has been passed to some other function so that it can be called later, even after f2() has completed. You do not want to have f2()'s stack frame around because that will keep all of f2()'s variables alive even if we are only interested in the free variables used by f3(). Cells hold on to the free variables so that the rest of f2() can be deallocated.

Lines 1217

This block represents the definition of f3(), which creates a local variable z. We then display w, x, y, z, all chased down from the innermost scope outward. The variable w cannot be found in f3(), f2(), or f1(), therefore, it is a global. The variable x is not found in f3() or f2(), so it is a closure variable from f1(). Similarly, y is a closure variable from f2(), and finally, z is local to f3().

Lines 3338

The rest of main() attempts to display closure variables for f1(), but it will never happen since there are no scopes in between the global scope and f1()'sthere is no scope that f1() can borrow from, ergo no closure can be createdso the conditional expression on line 34 will never evaluate to true. This code is just here for decorative purposes.

*Advanced Closures and Decorators Example

We saw a simple example of using closures and decorators in back in Section 11.3.6, deco.py. The following is a slightly more advanced example, to show you the real power of closures. The application "logs" function calls. The user chooses whether they want to log a function call before or after it has been invoked. If post-log is chosen, the execution time is also displayed.

Example 11.8. Logging Function Calls with Closures (funcLog.py)

This example shows a decorator that takes an argument that ultimately determines which closure will be used. Also featured is the power of closures.

1  #!/usr/bin/env python 2 3  from time import time 4 5  def logged(when): 6      def log(f, *args, **kargs): 7          print '''Called: 8  function: %s 9  args: %r 10 kargs: %r''' % (f, args, kargs) 11 12     def pre_logged(f): 13         def wrapper(*args, **kargs): 14             log(f, *args, **kargs) 15             return f(*args, **kargs) 16         return wrapper 17 18     def post_logged(f): 19         def wrapper(*args, **kargs): 20             now = time() 21             try: 22                 return f(*args, **kargs) 23             finally: 24                 log(f, *args, **kargs) 25                 print "time delta: %s" % (time()-now) 26         return wrapper 27 28     try: 29         return {"pre": pre_logged, 30              "post": post_logged}[when] 31     except KeyError, e: 32         raise ValueError(e), 'must be "pre" or "post"' 33 34  @logged("post") 35  def hello(name): 36      print "Hello,", name 37 38  hello("World!")

If you execute this script, you will get output similar to the following:

$ funcLog.py Hello, World! Called:     function: <function hello at 0x555f0>     args: ('World!',)     kargs: {}     time delta: 0.000471115112305

Line-by-Line Explanation

Lines 510, 2832

This body of code represents the core part of the logged() function, whose responsibility it is to take the user's request as to when the function call should be logged. Should it be before the target function is called or after? logged() has three helper inner functions defined within its definition: log(), pre_logged(), and post_logged().

log() is the function that does the actual logging. It just displays to standard output the name of the function and its arguments. If you were to use this function "in the real world," you would most likely send this output to a file, a database, or perhaps standard error (sys.stderr).

The last part of logged() in lines 28-32 is actually the first lines of code in the function that are not function declarations. It reads the user's selection when, and returns one of the *logged() functions so that it can then be called with the target function to wrap it.

Lines 1226

pre_logged() and post_logged() will both wrap the target function and log it in accordance with its name, e.g., post_logged() will log the function call after the target function has executed while pre_logged() does it before execution.

Depending on the user's selection, one of pre_logged() and post_logged() will be returned. When the decorator is called, it evaluates the decorator function first along with its argument. e.g., logged(when). Then the returned function object is called with the target function as its parameter, e.g., pre_logged(f) or post_logged(f).

Both *logged() functions include a closure named wrapper(). It calls the target function while logging it as appropriate. The functions return the wrapped function object, which then is reassigned to the original target function identifier.

Lines 3438

The main part of this script simply decorates the hello() function and executes it with the modified function object. When you call hello() on line 38, it is not the same as the function object that was created on line 35. The decorator on line 34 wraps the original function object with the specified decoration and returns a wrapped version of hello().

11.8.5. Scope and lambda

Python's lambda anonymous functions follow the same scoping rules as standard functions. A lambda expression defines a new scope, just like a function definition, so the scope is inaccessible to any other part of the program except for that local lambda/function.

Those lambda expressions declared local to a function are accessible only within that function; however, the expression in the lambda statement has the same scope access as the function. You can also think of a function and a lambda expression as siblings.

x = 10 def foo():     y = 5     bar = lambda :x+y     print bar()

We know that this code works fine now ...

>>> foo() 15

... however, we must again look to the past to see an extremely common idiom that was necessary to get code to work in older versions of Python. Before 2.1, we would get an error like what you see below because while the function and lambda have access to global variables, neither has access to the other's local scopes:

>>> foo() Traceback (innermost last):  File "<stdin>", line 1, in ?  File "<stdin>", line 4, in foo  File "<stdin>", line 3, in <lambda> NameError: y

In the example above, although the lambda expression was created in the local scope of foo(), it has access to only two scopes: its local scope and the global scope (also see Section 11.8.3). The solution was to add a variable with a default argument so that we could pass in a variable from an outer local scope to an inner one. In our example above, we would change the line with the lambda to look like this:

bar = lambda  y=y: x+y

With this change, it now works. The outer y's value will be passed in as an argument and hence the local y (local to the lambda function). You will see this common idiom all over Python code that you will come across; however, it still does not address the possibility of the outer y changing values, such as:

x = 10 def foo():     y = 5     bar = lambda  y=y: x+y     print bar()     y = 8     print bar()

The output is "totally wrong":

>>> foo() 15 15

The reason for this is that the value of the outer y was passed in and "set" in the lambda, so even though its value changed later on, the lambda definition did not. The only other alternative back then was to add a local variable z within the lambda expression that references the function local variable y.

x = 10 def foo():     y = 5     bar = lambda z:x+z     print bar(y)     y = 8     print bar(y)

All of this was necessary in order to get the correct output:

>>> foo() 15 18

This was also not preferred as now all places that call bar() would have to be changed to pass in a variable. Beginning in 2.1, the entire thing works perfectly without any modification:

x = 10 def foo():     y = 5     bar = lambda :x+y     print bar(y)     y = 8     print bar(y) >>> foo() 15 18

Are you not glad that "correct" statically nested scoping was (finally) added to Python? Many of the "old-timers" certainly are. You can read more about this important change in PEP 227.

11.8.6. Variable Scope and Namespaces

From our study in this chapter, we can see that at any given time, there are either one or two active scopesno more, no less. Either we are at the top-level of a module where we have access only to the global scope, or we are executing in a function where we have access to its local scope as well as the global scope. How do namespaces relate to scope?

From the Core Note in Section 11.8.1 we can also see that, at any given time, there are either two or three active namespaces. From within a function, the local scope encompasses the local namespace, the first place a name is searched for. If the name exists here, then checking the global scope (global and built-in namespaces) is skipped. From the global scope (outside of any function), a name lookup begins with the global namespace. If no match is found, the search proceeds to the built-in namespace.

We will now present Example 11.9, a script with mixed scope everywhere. We leave it as an exercise to the reader to determine the output of the program.

Example 11.9. Variable Scope (scope.py)

Local variables hide global variables, as indicated in this variable scope program. What is the output of this program? (And why?)

1  #!/usr/bin/env python 2  j, k = 1, 2 3 4  def proc1(): 5 6      j, k = 3, 4 7      print "j == %d and k == %d" % (j, k) 8      k = 5 9 10  def proc2(): 11 12      j = 6 13      proc1() 14      print "j == %d and k == %d" % (j, k) 15 16 17  k = 7 18  proc1() 19  print "j == %d and k == %d" % (j, k) 20 21  j = 8 22  proc2() 23  print "j == %d and k == %d" % (j, k)

Also see Section 12.3.1 for more on namespaces and variable scope.