Python functions beyond basics and a deep look at closure in Python.

All examples in this article use Python 3.3+.

### 1 Python functions beyond basics

Let's introduce several important features of Python functions first.

#### 1.1 Python function as a variable

Python function, essentially is also object, like a common Python variable, can be assigned to a variable.

Let's see a simple demo. First of all, define a random string. Then call `print`

and `len`

function and pass the defined string.

```
>>> slogan = "Life is short, I use Python"
>>> print
>>> <built-in function print>
>>> print(slogan)
>>> Life is short, I use Python
>>> len
>>> <built-in function len>
>>> len(slogan)
>>> 27
```

These work fine, nothing special.

Next step:

- assign
`print`

to a new variable`original_print`

- then assign
`len`

to`print`

.

```
>>> original_print = print # assign print to original_print
>>> original_print # original_print now becomes print
>>> <built-in function print>
>>> original_print(slogan)
>>> Life is short, I use Python
>>> print = len # assign len to print
>>> print # print now becomes len
>>> <built-in function len>
>>> print(slogan)
>>> 27
```

The conclusion here is straightforward: Python function can be assigned to a variable.

#### 1.2 Python function as function argument

A Python function can be passed as a argument to a function.

##### 1.2.1 Custom example

Let's give a `smart_add`

function as an example. It takes three arguments and the third argument is a function.

```
def smart_add(x, y, f):
return f(x) + f(y)
```

Try to call `smart_add`

and pass `abs`

function as the third argument to it.

```
>>> smart_add(-3, 7, abs) # abs(-3) + abs(7)
>>> 10
```

Again, try to pass another one: `math.sqrt`

.

```
>>> import math
>>> smart_add(4, 9, math.sqrt) # math.sqrt(4) + math.sqrt(9)
>>> 5.0
```

##### 1.2.2 Built-in example: `map`

```
>>> help(map)
# ...
map(func, *iterables) --> map object
# ...
```

According to the document: `map`

function will make an iterator that computes the function using arguments from each of the Iterables. Stops when the shortest Iterable is exhausted.

In short, `map`

will take each item `x`

in `iterables`

and map it to `func(x)`

.

`x in iterables |--map to--> func(x)`

The first parameter of `map`

is a function, and the second one is an Iterable collection. For example, pass `len`

function as the first argument and map string to its length.

```
>>> names = ["Tom", "Jerry", "Bugs Bunny"]
>>> mapped_obj = map(len, names)
>>> mapped_obj
>>> <map object at 0x102629320>
>>> print(list(mapped_obj))
[3, 5, 10]
```

Many other functions in Python are similar to `map`

, which takes a function as argument, such as `reduce`

, `filter`

.

You may have noticed that `map`

function can be replaced with list comprehensions.

```
# same effect as the map function
[func(item) for item in iterables]
```

In fact, the `reduce`

function was demoted from built-in in Python 2.x to the `functools`

module in Python 3 on that account. But the `map`

and `filter`

functions are still built-ins in Python 3.

Anyway, what we learned from this part is that Python function can be passed as an argument to a function.

#### 1.3 Return a function in a Python function

Let's define an `inner`

function within an `outer`

function and then return this `inner`

function from `outer`

.

```
def outer():
print('call outer() ...')
# define an inner function within the outer function
def inner():
print('call inner() ...')
# return the inner function
return inner
```

Call the `outer`

function and notice that the returned result is a function.

```
>>> r = outer() # call outer()
call outer() ...
>>> r # Returned result by calling outer() is a function
>>> <function outer.<locals>.inner at 0x1089c6d08>
>>> r() # Call the returned function
call inner() ...
```

One important thing to remember is not to confuse "return a function" with "return a data value".

```
import math
def demo_one():
return math.sqrt # return a function
def demo_two(x):
return math.sqrt(x) # return a data value
```

Look at another example below.

```
# pow_later.py
def pow_later(x):
y = 2
def lazy_pow():
print('calculate pow({}, {})...'.format(x, y))
return pow(x, y) # Use Python built-in function: pow
return lazy_pow
```

Try it in Python shell.

```
>>> from pow_later import pow_later
>>> my_pow = pow_later(3)
>>> my_pow
>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>
```

`pow_later`

returns a function that will actually calculate the result of `pow(3, 2)`

in the future.

So call it when you need, and you will get the real calculated result:

```
>>> my_pow()
calculate pow(3, 2)...
9
```

#### 1.4 Bonus: higher-order function and first-class function

A function that meet at least one of the following criteria is called a higher-order function.

- takes one or more functions as arguments
- returns a function as its result

In fact, A Python function is not only a higher-order function, but also a first-class function, which satisfies following four criteria:

- can be created at runtime
- can be assigned to a variable
- can be passed as a argument to a function
- can be returned as the result of a function

### 2 Python closure

Now take a deeper look at the latest example mentioned above.

```
def pow_later(x):
y = 2
def lazy_pow():
print('calculate pow({}, {})...'.format(x, y))
return pow(x, y)
return lazy_pow
```

We called `pow_later(3)`

and it returned a function object.

```
>>> my_pow = pow_later(3)
>>> my_pow
>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>
```

then we invoked the returned function object.

```
>>> my_pow()
calculate pow(3, 2)...
9
```

Obviously, the variable `y`

and the parameter `x`

are local variables of `pow_later`

function. So when `my_pow()`

was called, the `pow_later`

function had already returned, and its local variables also had gone. But in fact `my_pow()`

still remembered the values of `x`

and `y`

even the outer scope `pow_later`

was long gone. How did this happen?

#### 2.1 Free variable

If a variable in a function is neither a local variable nor a parameter of that function, this variable is called a free variable of that function.

In short, free variables are variables that are used locally, but defined in an enclosing scope.

In our case, `x`

is a parameter of `pow_later`

and `y`

is a local variable of `pow_later`

. But within `lazy_pow`

, `x`

and `y`

are free variables.

#### 2.2 Closure

##### 2.2.1 What is closure

Specifically speaking, `my_pow`

, actually the function object returned by calling `pow_later(x)`

, is a closure.

Note that the closure for `lazy_pow`

extends the scope of `lazy_pow`

function to include the binding for the free variables: `x`

and `y`

.

Generally speaking, a closure is a structure (code blocks, function object, callable object, etc.) storing a function together with an environment. The **environment** here means information about free variables that function bounded, especially values or storage locations of free variables.

For example, a closure is created, returned and assigned to `my_pow`

after following function call.

`>>> my_pow = pow_later(3)`

Essentially, this closure is the codes of function `lazy_pow`

together with free variables `x`

and `y`

.

##### 2.2.2 Inspect closure

You can see that the closure keeps names of free variables by inspecting `__code__`

attribute of `my_pow`

function which represents the compiled body of the function.

```
>>> my_pow.__code__.co_freevars
>>> ('x', 'y')
```

Meanwhile, `pow_later`

will also keep names of local variables that are referenced by its nested functions in `co_cellvars`

attribute of its code object.

```
>>> pow_later.__code__.co_cellvars
>>> ('x', 'y')
```

However, where is the values of free variables?

```
>>> dir(my_pow)
>>> my_pow.__closure__
>>> (<cell at 0x10a428348: int object at 0x109e06b60>, <cell at 0x10a428378: int object at 0x109e06b40>)
```

Note that `my_pow`

has an attribute named `__closure__`

and it's a tuple with two elements.

```
>>> dir(my_pow.__closure__[0])
>>> ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']
>>> my_pow.__closure__[0].cell_contents
>>> 3
>>> my_pow.__closure__[1].cell_contents
>>> 2
```

So `__closure__`

is a tuple of `cells`

that contain bounded values of free variables.

If your Python version is 3.3+, you can use `inspect`

module to inspect. The `nonlocals`

dictionary in inspecting result is exactly the bounded free variables and their values.

```
>>> import inspect
>>> inspect.getclosurevars(my_pow)
ClosureVars(nonlocals={'x': 3, 'y': 2}, globals={}, builtins={'print': <built-in function print>, 'pow': <built-in function pow>, 'format': <built-in function format>}, unbound=set())
```

##### 2.2.3 **closure**

Functions without free variables are not closures.

```
def f(x):
def g():
pass
return g
```

Note that returned function `g`

has no free variable. And its `__closure__`

is `None`

.

```
>>> h=f(1)
>>> h
>>> <function f.<locals>.g at 0x10f650158>
>>> h.__code__.co_freevars
>>> ()
>>> print(h.__closure__)
>>> None
```

Global variables are not free variables in Python. So global functions are not closures.

```
>>> data=200 # global
>>> def d(): # global
>>> print(data)
...
...
>>> d()
>>> 200
>>> d.__code__.co_freevars
>>> ()
>>> print(d.__closure__)
>>> None
```

`__closure__`

attribute of global functions is `None`

.

##### 2.2.4 nonlocal declaration

Let's review our `pow_later(x)`

function.

- pass a number
`x`

to function`pow_later`

; `pow_later`

will return a function object;- the returned function object
`my_pow`

will calculate`x**2`

(`y=2`

) each time it is called.

Now I'd like to change above behavior, let `y`

increase 1 automatically each time `my_pow`

is called. That is:

- the first time call, calculate
`x**2`

; - the second time call, calculate
`x**3`

; - the third time call, calculate
`x**4`

; - ....

The updated source codes are as follows.

```
# pow_later.py
def pow_later(x):
y = 2
def lazy_pow():
print('calculate pow({}, {})...'.format(x, y))
result = pow(x, y)
y = y + 1 # increase y
return result
return lazy_pow
```

Try it in Python shell.

```
>>> from pow_later import pow_later
>>> my_pow = pow_later(3)
>>> my_pow
>>> <function pow_later.<locals>.lazy_pow at 0x108e020d0>
```

So far so good, let's call `my_pow`

to see result.

```
>>> my_pow()
>>> Traceback (most recent call last):
...
UnboundLocalError: local variable 'y' referenced before assignment
```

The error message is clear enough.

- It's a
`UnboundLocalError`

`y`

is a local variable- local variable
`y`

referenced before assignment

The problem happens in this line: `y = y + 1`

.

We are actually assigning to `y`

in `lazy_pow`

scope, and that makes `y`

becomes local to `lazy_pow`

scope. So Python considers `y`

a local variable of `lazy_pow`

. Before assigning to that local variable, Python will first read the local variable `y`

. But `y`

is a free variable as mentioned earlier and there is no local variable named `y`

in `lazy_pow`

scope at all.

You may think, OK, we don't assign! How about use `y += 1`

instead of `y = y + 1`

? The `+=`

operation is performed in-place, meaning that rather than creating and assigning a new value to the variable, the old variable is modified instead.

The answer is: no change here. Because `y`

is a number, which is an immutable type. `+=`

will also create a new number object with new value behind the scene and **assign** the reference of the new object to `y`

.

To deal with this situation, a `nonlocal`

declaration was introduced in Python 3. It marks a variable as a free variable even though it is assigned a new value within the function.

```
# pow_later.py
def pow_later(x):
y = 2
def lazy_pow():
nonlocal y # nonlocal declaration
print('calculate pow({}, {})...'.format(x, y))
result = pow(x, y)
y = y + 1
return result
return lazy_pow
```

Now the closure works well.

```
>>> from pow_later import pow_later
>>> my_pow = pow_later(3)
>>> my_pow()
>>> calculate pow(3, 2)...
9
>>> my_pow()
>>> calculate pow(3, 3)...
27
>>> my_pow()
>>> calculate pow(3, 4)...
81
```

### 3 Summary

Two topics were discussed in this article.

First, Python functions are first-class functions.

Second, what is closure and how it works in Python.