Section 22.2. Extending Python by Writing Extensions


22.2. Extending Python by Writing Extensions

Creating extensions for Python involves three main steps:

  1. Creating application code

  2. Wrapping code with boilerplates

  3. Compilation and testing

In this section, we will break out all three pieces and expose them all to you.

22.2.1. Create Your Application Code

First, before any code becomes an extension, create a standalone "library." In other words, create your code keeping in mind that it is going to turn into a Python module. Design your functions and objects with the vision that Python code will be communicating and sharing data with your C code and vice versa.

Next, create test code to bulletproof your software. You may even use the "Pythonic" development method of designating your main() function in C as the testing application so that if your code is compiled, linked, and loaded into an executable (as opposed to just a shared object), invocation of such an executable will result in a regression test of your software library. For our extension example below, this is exactly what we do.

The test case involves two C functions that we want to bring to the world of Python programming. The first is the recursive factorial function, fac(). The second, reverse(), is a simple string reverse algorithm, whose main purpose is to reverse a string "in place," that is, to return a string whose characters are all reversed from their original positions, all without allocating a separate string to copy in reverse order. Because this involves the use of pointers, we need to carefully design and debug our code before bringing Python into the picture.

Our first version, Extest1.c, is presented in Example 22.1.

This code consists of a pair of functions, fac() and reverse(), which are implementations of the functionality we described above. fac() takes a single integer argument and recursively calculates the result, which is eventually returned to the caller once it exits the outermost call.

Example 22.1. Pure C Version of Library (Extest1.c)

The following code represents our library of C functions which we want to wrap so that we can use this code from within the Python interpreter. main() is our tester function.

1    #include <stdio.h> 2    #include <stdlib.h> 3    #include <string.h> 4 5    int fac(int n) 6    { 7       if (n < 2) return(1); /* 0! == 1! == 1 */ 8       return (n)*fac(n-1); /* n! == n*(n-1)! */ 9    } 10 11   char *reverse(char *s) 12   { 13       register char t,                    /* tmp */ 14               *p = s,                     /* fwd */ 15               *q = (s + (strlen(s)-1));   /* bwd */ 16 17       while (p < q)            /* if p < q */ 18         {     /*swap & mv ptrs */ 19             t = *p; 20             *p++ = *q; 21             *q-- = t; 22         } 23         return s; 24   } 25 26   int main() 27   { 28        char s[BUFSIZ]; 29        printf("4! == %d\n", fac(4)); 30        printf("8! == %d\n", fac(8)); 31        printf("12! == %d\n", fac(12)); 32        strcpy(s, "abcdef"); 33        printf("reversing 'abcdef', we get '%s'\n", \ 34            reverse(s)); 35        strcpy(s, "madam"); 36        printf("reversing 'madam', we get '%s'\n", \ 37            reverse(s)); 38        return 0; 39   }

The last piece of code is the required main() function. We use it to be our tester, sending various arguments to fac() and reverse() . With this function, we can actual tell whether our code works (or not).

Now we should compile the code. For many versions of Unix with the gcc compiler, we can use the following command:

$ gcc Extest1.c -o Extest $


To run our program, we issue the following command and get the output:

$ Extest 4! == 24 8! == 40320 12! == 479001600 reversing 'abcdef', we get 'fedcba' reversing 'madam', we get 'madam' $


We stress again that you should try to complete your code as much as possible, because you do not want to mix debugging of your library with potential bugs when integrating with Python. In other words, keep the debugging of your core code separate from the debugging of the integration. The closer you write your code to Python interfaces, the sooner your code will be integrated and work correctly.

Each of our functions takes a single value and returns a single value. It's pretty cut and dried, so there shouldn't be a problem integrating with Python. Note that, so far, we have not seen any connection or relationship with Python. We are simply creating a standard C or C++ application.

22.2.2. Wrap Your Code in Boilerplate

The entire implementation of an extension primarily revolves around the "wrapping" concept that we introduced earlier in Section 13.15.1. You should design your code in such a way that there is a smooth transition between the world of Python and your implementing language. This interfacing code is commonly called "boilerplate" code because it is a necessity if your code is to talk to the Python interpreter.

There are four main pieces to the boilerplate software:

  1. Include Python header file

  2. Add PyObject* Module_func() Python wrappers for each module function

  3. Add PyMethodDef Module Methods[] array/table for each module function

  4. Add void init Module() module initializer function

Include Python Header File

The first thing you should do is to find your Python include files and make sure your compiler has access to that directory. On most Unix-based systems, this would be either /usr/local/include/python2.x or /usr/include/python2.x, where the "2.x" is your version of Python. If you compiled and installed your Python interpreter, you should not have a problem because the system generally knows where your files are installed.

Add the inclusion of the Python.h header file to your source. The line will look something like:

         #include "Python.h"


That is the easy part. Now you have to add the rest of the boilerplate software.

Add PyObject* Module _func() Python Wrappers for Each Function

This part is the trickiest. For each function you want accessible to the Python environment, you will create a static PyObject* function with the module name along with an underscore ( _ ) prepended to it.

For example, we want fac() to be one of the functions available for import from Python and we will use Extest as the name of our final module, so we create a "wrapper" called Extest_fac(). In the client Python script, there will be an "import Extest" and an "Extest.fac()" call somewhere (or just "fac()" for "from Extest import fac").

The job of the wrapper is to take Python values, convert them to C, then make a call to the appropriate function with what we want. When our function has completed, and it is time to return to the world of Python, it is also the job of this wrapper to take whatever return values we designate, convert them to Python, and then perform the return, passing back any values as necessary.

In the case of fac(), when the client program invokes Extest.fac(), our wrapper will be called. We will accept a Python integer, convert it to a C integer, call our C function fac() and obtain another integer result. We then have to take that return value, convert it back to a Python integer, then return from the call. (In your head, try to keep in mind that you are writing the code that will proxy for a "def fac(n)" declaration. When you are returning, it is as if that imaginary Python fac() function is completing.)

So, you're asking, how does this conversion take place? The answer is with the PyArg_Parse*() functions when going from Python to C, and Py_BuildValue() when returning from C to Python.

The PyArg_Parse*() functions are similar to the C sscanf() function. It takes a stream of bytes, and, according to some format string, parcels them off to corresponding container variables, which, as expected, take pointer addresses. They both return 1 on successful parsing and 0 otherwise.

Py_BuildValue() works like sprintf(), taking a format string and converting all arguments to a single returned object containing those values in the formats that you requested.

You will find a summary of these functions in Table 22.1.

Table 22.1. Converting Data Between Python and C/C++

Function

Description

Python to C

 

int PyArg_ParseTuple()

Converts (a tuple of) arguments passed from Python to C

int PyArg_ParseTupleAndKeywords()

Same as PyArg_ParseTuple() but also parses keyword arguments

C to Python

 

PyObject* Py_BuildValue()

Converts C data values into a Python return object, either a single object or a single tuple of objects


A set of conversion codes is used to convert data objects between C and Python; they are given in Table 22.2.

Table 22.2. Common Codes to Convert Data Between Python and C/C++

Format Code

Python Type

C/C++ Type

s

str

char*

z

str/None

char* /NULL

i

int

int

l

long

long

c

str

char

d

float

double

D

complex

Py_Complex*

O

(any)

PyObject*

S

str

PyStringObject


These conversion codes are the ones given in the respective format strings that dictate how the values should be converted when moving between both languages. Note: The conversion types are different for Java since all data types are classes. Consult the Jython documentation to obtain the corresponding Java types for Python objects. The same applies for C# and VB.NET.

Here we show you our completed Extest_fac() wrapper function:

static PyObject * Extest_fac(PyObject *self, PyObject *args) {      int res;              // parse result      int num;              // arg for fac()      PyObject* retval;     // return value      res = PyArg_ParseTuple(args, "i", &num);      if (!res) {           // TypeError          return NULL;      }      res = fac(num);      retval = (PyObject*)Py_BuildValue("i", res);      return retval; }


The first step is to parse the data received from Python. It should be a regular integer, so we use the "i" conversion code to indicate as such. If the value was indeed an integer, then it gets stored in the num variable. Otherwise, PyArg_ParseTuple() will return a NULL, in which case we also return one. In our case, it will generate a TypeError exception that tells the client user that we are expecting an integer.

We then call fac() with the value stored in num and put the result in res, reusing that variable. Now we build our return object, a Python integer, again using a conversion code of "i." Py_BuildValue() creates an integer Python object which we then return. That's all there is to it!

In fact, once you have created wrapper after wrapper, you tend to shorten your code somewhat to avoid extraneous use of variables. Try to keep your code legible, though. We take our Extest_fac() function and reduce it to its smaller version given here, using only one variable, num:

static PyObject * Extest_fac(PyObject *self, PyObject *args) {     int num;     if (!PyArg_ParseTuple(args, "i", &num))            return NULL;     return (PyObject*)Py_BuildValue("i", fac(num)); }


What about reverse()? Well, since you already know how to return a single value, we are going to change our reverse() example somewhat, returning two values instead of one. We will return a pair of strings as a tuple, the first element being the string as passed in to us, and the second being the newly reversed string.

To show you that there is some flexibility, we will call this function Extest.doppel() to indicate that its behavior differs from reverse(). Wrapping our code into an Extest_doppel() function, we get:

static PyObject * Extest_doppel(PyObject *self, PyObject *args) {     char *orig_str;     if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL;     return (PyObject*)Py_BuildValue("ss", orig_str, \         reverse(strdup(orig_str))); }


As in Extest_fac(), we take a single input value, this time a string, and store it into orig_str. Notice that we use the "s" conversion code now. We then call strdup() to create a copy of the string. (Since we want to return the original one as well, we need a string to reverse, so the best candidate is just a copy of the string.) strdup() creates and returns a copy, which we immediate dispatch to reverse(). We get back a reversed string.

As you can see, Py_BuildValue() puts together both strings using a conversion string of "ss." This creates a tuple of two strings, the original string and the reversed one. End of story, right? Unfortunately, no.

We got caught by one of the perils of C programming: the memory leak, that is, when memory is allocated but not freed. Memory leaks are analogous to borrowing books from the library but not returning them. You should always release resources that you have acquired when you no longer require them. How did we commit such a crime with our code (which looks innocent enough)?

When Py_BuildValue() puts together the Python object to return, it makes copies of the data it has been passed. In our case here, that would be a pair of strings. The problem is that we allocated the memory for the second string, but we did not release that memory when we finished, leaking it. What we really want to do is to build the return object and then free the memory that we allocated in our wrapper. We have no choice but to lengthen our code to:

static PyObject * Extest_doppel(PyObject *self, PyObject *args) {     char *orig_str;                 // original string     char *dupe_str;                 // reversed string     PyObject* retval;     if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL;     retval = (PyObject*)Py_BuildValue("ss", orig_str, \         dupe_str=reverse(strdup(orig_str)));     free(dupe_str);     return retval; }


We introduce the dupe_str variable to point to the newly allocated string and build the return object. Then we free() the memory allocated and finally return back to the caller. Now we are done.

Add PyMethodDef ModuleMethods[] Array/Table for Each Module Function

Now that both of our wrappers are complete, we want to list them somewhere so that the Python interpreter knows how to import and access them. This is the job of theModuleMethods[] array.

It is made up of an array of arrays, with each individual array containing information about each function, terminated by a NULL array marking the end of the list. For our Extest module, we create the following ExtestMethods[] array:

static PyMethodDef ExtestMethods[] = {      { "fac", Extest_fac, METH_VARARGS },      { "doppel", Extest_doppel, METH_VARARGS },      { NULL, NULL }, };


The Python-accessible names are given, followed by the corresponding wrapping functions. The constant METH_VARARGS is given, indicating a set of arguments in the form of a tuple. If we are using PyArg_ParseTupleAndKeywords() with keyworded arguments, we would logically OR this flag with the METH_KEYWORDS constant. Finally, a pair of NULLs properly terminates our list of two functions.

Add void initModule() Module Initializer Function

The final piece to our puzzle is the module initializer function. This code is called when our module is imported for use by the interpreter. In this code, we make one call to Py_InitModule () along with the module name and the name of the ModuleMethods[] array so that the interpreter can access our module functions. For our Extest module, our initExtest() procedure looks like this:

void initExtest() {          Py_InitModule("Extest", ExtestMethods); }


We are now done with all our wrapping. We add all this code to our original code from Extest1.c and merge the results into a new file called Extest2.c, concluding the development phase of our example.

Another approach to creating an extension would be to make your wrapping code first, using "stubs" or test or dummy functions which will, during the course of development, be replaced by the fully functional pieces of implemented code. That way you can ensure that your interface between Python and C is correct, and then use Python to test your C code.

22.2.3. Compilation

Now we are on to the compilation phase. In order to get your new wrapper Python extension to build, you need to get it to compile with the Python library. This task has been standardized (since 30) across platforms to make life a lot easier for extension writers. The distutils package is used to build, install, and distribute modules, extensions, and packages. It came about back in Python 2.0 and replaced the old 1.x way of building extensions using "makefiles." Using distutils, we can follow this easy recipe:

  1. Create setup.py

  2. Compile and link your code by running setup.py

  3. Import your module from Python

  4. Test function

Create setup.py

The next step is to create a setup.py file. The bulk of the work will be done by the setup() function. All the lines of code that come before that call are preparatory steps. For building extension modules, you need to create an Extension instance per extension. Since we only have one, we only need one Extension instance:

    Extension('Extest', sources=['Extest2.c'])


The first argument is the (full) extension name, including any high-level packages if necessary. The name should be in full dotted-attribute notation. Ours is standalone, hence the name "Extest." sources is a list of all the source files. Again, we only have the one, Extest2.c.

Now we are ready to call setup(). It takes a name argument for what it is building and a list of the items to build. Since we are creating an extension, we set it a list of extension modules to build as ext_modules. The syntax will be like this:

    setup('Extest', ext_modules=[...])


Since we only have one module, we combine the instantiation of our extension module into our call to setup(), setting the module name as "constant" MOD on the preceding line:

   MOD = 'Extest'    setup(name=MOD, ext_modules=[        Extension(MOD, sources=['Extest2.c'])])


There are many more options to setup(), which are too numerous to list here. You can find out more about creating setup.py and calling setup() in the official Python documentation that we refer to at the end of this chapter. Example 22.2 shows the complete script that we are using for our example.

Example 22.2. The Build Script (setup.py)

This script compiles our extension into the build/lib.* subdirectory.

   1    #!/usr/bin/env python    2    3    from distutils.core import setup, Extension    4    5    MOD = 'Extest'    6    setup(name=MOD, ext_modules=[    7        Extension(MOD, sources=['Extest2.c'])])

Compile and Link Your Code by Running setup.py

Now that we have our setup.py file, we can build our extension by running it with the "build" directive, as we have done here on our Mac (your output will differ based on the version of the operating system you are running as well as the version of Python you are using):

 $ python setup.py build  running build  running build_ext  building 'Extest' extension  creating build  creating build/temp.macosx-10.x-fat-2.x  gcc -fno-strict-aliasing -Wno-long-double -no-cpp-  precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g  -I/usr/include -I/usr/local/include -I/sw/include -I/  usr/local/include/python2.x -c Extest2.c -o build/  temp.macosx-10.x-fat-2.x/Extest2.o  creating build/lib.macosx-10.x-fat-2.x  gcc -g -bundle -undefined dynamic_lookup -L/usr/lib -L/  usr/local/lib -L/sw/lib -I/usr/include -I/usr/local/  include -I/sw/include build/temp.macosx-10.x-fat-2.x/  Extest2.o -o build/lib.macosx-10.x-fat-2.x/Extest.so


22.2.4. Import and Test

Import Your Module from Python

Your extension module will be created in the build/lib.* directory from where you ran your setup.py script. You can either change to that directory to test your module or install it into your Python distribution with:

 $ python setup.py install


If you do install it, you will get the following output:

 running install  running build  running build_ext  running install_lib  copying build/lib.macosx-10.x-fat-2.x/Extest.so ->  /usr/local/lib/python2.x/site-packages


Now we can test out our module from the interpreter:

 >>> import Extest  >>> Extest.fac(5)  120  >>> Extest.fac(9)  362880  >>> Extest.doppel('abcdefgh')  ('abcdefgh', 'hgfedcba')  >>> Extest.doppel("Madam, I'm Adam.")  ("Madam, I'm Adam.", ".madA m'I, madaM")


Test Function

The one last thing we want to do is to add a test function. In fact, we already have one, in the form of the main() function. Now, it is potentially dangerous to have a main() function in our code because there should only be one main() in the system. We remove this danger by changing the name of our main() to test() and wrapping it, adding Extest_test() and updating the ExtestMethods array so that they both look like this:

static PyObject * Extest_test(PyObject *self, PyObject *args) {     test();     return (PyObject*)Py_BuildValue(""); } static PyMethodDef ExtestMethods[] = {     { "fac", Extest_fac, METH_VARARGS },     { "doppel", Extest_doppel, METH_VARARGS },     { "test", Extest_test, METH_VARARGS },     { NULL, NULL }, };


The Extest_test() module function just runs test() and returns an empty string, resulting in a Python value of None being returned to the caller.

Now we can run the same test from Python:

   >>> Extest.test()    4! == 24    8! == 40320    12! == 479001600    reversing 'abcdef', we get 'fedcba'    reversing 'madam', we get 'madam'    >>>


In Example 22.3, we present the final version of Extest2.c that was used to generate the output we just witnessed.

Example 22.3. Python-Wrapped Version of C Library (Extest2.c)

 1  #include <stdio.h> 2  #include <stdlib.h> 3  #include <string.h> 4 5  int fac(int n) 6  { 7      if (n < 2) return(1); 8      return (n)*fac(n-1); 9  } 10 11 char *reverse(char *s) 12 { 13     register char t, 14                 *p = s, 15                 *q = (s + (strlen(s) - 1)); 16 17     while (s && (p < q)) 18     { 19         t = *p; 20         *p++ = *q; 21         *q-- = t; 22     } 23     return s; 24 } 25 26 int test() 27 { 28      char s[BUFSIZ]; 29      printf("4! == %d\n", fac(4)); 30      printf("8! == %d\n", fac(8)); 31      printf("12! == %d\n", fac(12)); 32      strcpy(s, "abcdef"); 33      printf("reversing 'abcdef', we get '%s'\n", \ 34          reverse(s)); 35      strcpy(s, "madam"); 36      printf("reversing 'madam', we get '%s'\n", \ 37          reverse(s)); 38      return 0; 39 } 40 41 #include "Python.h" 42 43 static PyObject * 44 Extest_fac(PyObject *self, PyObject *args) 45 { 46     int num; 47     if (!PyArg_ParseTuple(args, "i", &num)) 48         return NULL; 49     return (PyObject*)Py_BuildValue("i", fac(num));} 50 } 51 52 static PyObject * 53 Extest_doppel(PyObject *self, PyObject *args) 54 { 55     char  *orig_str; 56     char *dupe_str; 57     PyObject* retval; 58 59     if (!PyArg_ParseTuple(args, "s", &orig_str)) 60         return NULL; 61     retval = (PyObject*)Py_BuildValue("ss", orig_str, \ 62         dupe_str=reverse(strdup(orig_str))); 63     free(dupe_str); 64     return retval; 65 } 66 67 static PyObject * 68 Extest_test(PyObject *self, PyObject *args) 69 { 70     test(); 71     return (PyObject*)Py_BuildValue(""); 72 } 73 74 static PyMethodDef 75 ExtestMethods[] = 76 { 77     { "fac", Extest_fac, METH_VARARGS }, 78     { "doppel", Extest_doppel, METH_VARARGS }, 79     { "test", Extest_test, METH_VARARGS }, 80     { NULL, NULL }, 81 }; 82 83 void initExtest() 84 { 85     Py_InitModule("Extest", ExtestMethods); 86 }

In this example, we chose to segregate our C code from our Python code. It just kept things easier to read and is no problem with our short example. In practice, these source files tend to get large, and some choose to implement their wrappers completely in a different source file, i.e., ExtestWrappers.c or something of that nature.

22.2.5. Reference Counting

You may recall that Python uses reference counting as a means of keeping track of objects and deallocating objects no longer referenced as part of the garbage collection mechanism. When creating extensions, you must pay extra special attention to how you manipulate Python objects because you must be mindful of whether or not you need to change the reference count for such objects.

There are two types of references you may have to an object, one of which is an owned reference, meaning that the reference count to the object is incremented by one to indicate your ownership. One place where you would definitely have an owned reference is where you create a Python object from scratch.

When you are done with a Python object, you must dispose of your ownership, either by decrementing the reference count, transferring your ownership by passing it on, or storing the object. Failure to dispose of an owned reference creates a memory leak.

You may also have a borrowed reference to an object. Somewhat lower on the responsibility ladder, this is where you are passed the reference of an object, but otherwise do not manipulate the data in any way. Nor do you have to worry about its reference count, as long as you do not hold on to this reference after its reference count has decreased to zero. You may convert your borrowed reference to an owned reference simply by incrementing an object's reference count.

Python provides a pair of C macros which are used to change the reference count to a Python object. They are given in Table 22.3.

Table 22.3. Macros for Performing Python Object Reference Counting

Function

Description

Py_INCREF(obj)

Increment the reference count to obj

Py_DECREF(obj)

Decrement the reference count to obj


In our above Extest_test() function, we return None by building a PyObject with an empty string; however, it can also be accomplished by becoming an owner of the None object, PyNone, incrementing your reference count to it, and returning it explicitly, as in the following alternative piece of code:

 static PyObject *  Extest_test(PyObject *self, PyObject *args) {           test();           Py_INCREF(Py_None);           return PyNone;  }


Py_INCREF() and Py_DECREF() also have versions that check for NULL objects. They are Py_XINCREF() and Py_XDECREF(), respectively.

We strongly urge the reader to consult the Python documentation regarding extending and embedding Python for all the details with regard to reference counting (see the documentation reference in the Appendix).

22.2.6. Threading and the GIL

Extension writers must be aware that their code may be executed in a multithreaded Python environment. Back in Section 18.3.1, we introduced the Python Virtual Machine (PVM) and the Global Interpreter Lock (GIL) and described how only one thread of execution can be running at any given time in the PVM and that the GIL is responsible for keeping other threads from running. Furthermore, we indicated that code calling external functions such as in extension code would keep the GIL locked until the call returns.

We also hinted that there was a remedy, a way for the extension programmer to release the GIL, for example before performing a system call. This is accomplished by "blocking" your code off to where threads may (and may not) run safely using another pair of C macros, Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. A block of code bounded by these macros will permit other threads to run.

As with the reference counting macros, we urge you to consult with the documentation regarding extending and embedding Python as well as the Python/C API reference manual.



Core Python Programming
Core Python Programming (2nd Edition)
ISBN: 0132269937
EAN: 2147483647
Year: 2004
Pages: 334
Authors: Wesley J Chun

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net