22.2. Extending Python by Writing ExtensionsCreating extensions for Python involves three main steps:
In this section, we will break out all three pieces and expose them all to you. 22.2.1. Create Your Application CodeFirst, before any code becomes an extension, create a standalone "library." In other words, create your code keeping in mind that it is going to turn into a Python module. Design your functions and objects with the vision that Python code will be communicating and sharing data with your C code and vice versa. Next, create test code to bulletproof your software. You may even use the "Pythonic" development method of designating your main() function in C as the testing application so that if your code is compiled, linked, and loaded into an executable (as opposed to just a shared object), invocation of such an executable will result in a regression test of your software library. For our extension example below, this is exactly what we do. The test case involves two C functions that we want to bring to the world of Python programming. The first is the recursive factorial function, fac(). The second, reverse(), is a simple string reverse algorithm, whose main purpose is to reverse a string "in place," that is, to return a string whose characters are all reversed from their original positions, all without allocating a separate string to copy in reverse order. Because this involves the use of pointers, we need to carefully design and debug our code before bringing Python into the picture. Our first version, Extest1.c, is presented in Example 22.1. This code consists of a pair of functions, fac() and reverse(), which are implementations of the functionality we described above. fac() takes a single integer argument and recursively calculates the result, which is eventually returned to the caller once it exits the outermost call. Example 22.1. Pure C Version of Library (Extest1.c)
The last piece of code is the required main() function. We use it to be our tester, sending various arguments to fac() and reverse() . With this function, we can actual tell whether our code works (or not). Now we should compile the code. For many versions of Unix with the gcc compiler, we can use the following command: $ gcc Extest1.c -o Extest $ To run our program, we issue the following command and get the output: $ Extest 4! == 24 8! == 40320 12! == 479001600 reversing 'abcdef', we get 'fedcba' reversing 'madam', we get 'madam' $ We stress again that you should try to complete your code as much as possible, because you do not want to mix debugging of your library with potential bugs when integrating with Python. In other words, keep the debugging of your core code separate from the debugging of the integration. The closer you write your code to Python interfaces, the sooner your code will be integrated and work correctly. Each of our functions takes a single value and returns a single value. It's pretty cut and dried, so there shouldn't be a problem integrating with Python. Note that, so far, we have not seen any connection or relationship with Python. We are simply creating a standard C or C++ application. 22.2.2. Wrap Your Code in BoilerplateThe entire implementation of an extension primarily revolves around the "wrapping" concept that we introduced earlier in Section 13.15.1. You should design your code in such a way that there is a smooth transition between the world of Python and your implementing language. This interfacing code is commonly called "boilerplate" code because it is a necessity if your code is to talk to the Python interpreter. There are four main pieces to the boilerplate software:
Include Python Header FileThe first thing you should do is to find your Python include files and make sure your compiler has access to that directory. On most Unix-based systems, this would be either /usr/local/include/python2.x or /usr/include/python2.x, where the "2.x" is your version of Python. If you compiled and installed your Python interpreter, you should not have a problem because the system generally knows where your files are installed. Add the inclusion of the Python.h header file to your source. The line will look something like: #include "Python.h" That is the easy part. Now you have to add the rest of the boilerplate software. Add PyObject* Module _func() Python Wrappers for Each FunctionThis part is the trickiest. For each function you want accessible to the Python environment, you will create a static PyObject* function with the module name along with an underscore ( _ ) prepended to it. For example, we want fac() to be one of the functions available for import from Python and we will use Extest as the name of our final module, so we create a "wrapper" called Extest_fac(). In the client Python script, there will be an "import Extest" and an "Extest.fac()" call somewhere (or just "fac()" for "from Extest import fac"). The job of the wrapper is to take Python values, convert them to C, then make a call to the appropriate function with what we want. When our function has completed, and it is time to return to the world of Python, it is also the job of this wrapper to take whatever return values we designate, convert them to Python, and then perform the return, passing back any values as necessary. In the case of fac(), when the client program invokes Extest.fac(), our wrapper will be called. We will accept a Python integer, convert it to a C integer, call our C function fac() and obtain another integer result. We then have to take that return value, convert it back to a Python integer, then return from the call. (In your head, try to keep in mind that you are writing the code that will proxy for a "def fac(n)" declaration. When you are returning, it is as if that imaginary Python fac() function is completing.) So, you're asking, how does this conversion take place? The answer is with the PyArg_Parse*() functions when going from Python to C, and Py_BuildValue() when returning from C to Python. The PyArg_Parse*() functions are similar to the C sscanf() function. It takes a stream of bytes, and, according to some format string, parcels them off to corresponding container variables, which, as expected, take pointer addresses. They both return 1 on successful parsing and 0 otherwise. Py_BuildValue() works like sprintf(), taking a format string and converting all arguments to a single returned object containing those values in the formats that you requested. You will find a summary of these functions in Table 22.1.
A set of conversion codes is used to convert data objects between C and Python; they are given in Table 22.2.
These conversion codes are the ones given in the respective format strings that dictate how the values should be converted when moving between both languages. Note: The conversion types are different for Java since all data types are classes. Consult the Jython documentation to obtain the corresponding Java types for Python objects. The same applies for C# and VB.NET. Here we show you our completed Extest_fac() wrapper function: static PyObject * Extest_fac(PyObject *self, PyObject *args) { int res; // parse result int num; // arg for fac() PyObject* retval; // return value res = PyArg_ParseTuple(args, "i", &num); if (!res) { // TypeError return NULL; } res = fac(num); retval = (PyObject*)Py_BuildValue("i", res); return retval; } The first step is to parse the data received from Python. It should be a regular integer, so we use the "i" conversion code to indicate as such. If the value was indeed an integer, then it gets stored in the num variable. Otherwise, PyArg_ParseTuple() will return a NULL, in which case we also return one. In our case, it will generate a TypeError exception that tells the client user that we are expecting an integer. We then call fac() with the value stored in num and put the result in res, reusing that variable. Now we build our return object, a Python integer, again using a conversion code of "i." Py_BuildValue() creates an integer Python object which we then return. That's all there is to it! In fact, once you have created wrapper after wrapper, you tend to shorten your code somewhat to avoid extraneous use of variables. Try to keep your code legible, though. We take our Extest_fac() function and reduce it to its smaller version given here, using only one variable, num: static PyObject * Extest_fac(PyObject *self, PyObject *args) { int num; if (!PyArg_ParseTuple(args, "i", &num)) return NULL; return (PyObject*)Py_BuildValue("i", fac(num)); } What about reverse()? Well, since you already know how to return a single value, we are going to change our reverse() example somewhat, returning two values instead of one. We will return a pair of strings as a tuple, the first element being the string as passed in to us, and the second being the newly reversed string. To show you that there is some flexibility, we will call this function Extest.doppel() to indicate that its behavior differs from reverse(). Wrapping our code into an Extest_doppel() function, we get: static PyObject * Extest_doppel(PyObject *self, PyObject *args) { char *orig_str; if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL; return (PyObject*)Py_BuildValue("ss", orig_str, \ reverse(strdup(orig_str))); } As in Extest_fac(), we take a single input value, this time a string, and store it into orig_str. Notice that we use the "s" conversion code now. We then call strdup() to create a copy of the string. (Since we want to return the original one as well, we need a string to reverse, so the best candidate is just a copy of the string.) strdup() creates and returns a copy, which we immediate dispatch to reverse(). We get back a reversed string. As you can see, Py_BuildValue() puts together both strings using a conversion string of "ss." This creates a tuple of two strings, the original string and the reversed one. End of story, right? Unfortunately, no. We got caught by one of the perils of C programming: the memory leak, that is, when memory is allocated but not freed. Memory leaks are analogous to borrowing books from the library but not returning them. You should always release resources that you have acquired when you no longer require them. How did we commit such a crime with our code (which looks innocent enough)? When Py_BuildValue() puts together the Python object to return, it makes copies of the data it has been passed. In our case here, that would be a pair of strings. The problem is that we allocated the memory for the second string, but we did not release that memory when we finished, leaking it. What we really want to do is to build the return object and then free the memory that we allocated in our wrapper. We have no choice but to lengthen our code to: static PyObject * Extest_doppel(PyObject *self, PyObject *args) { char *orig_str; // original string char *dupe_str; // reversed string PyObject* retval; if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL; retval = (PyObject*)Py_BuildValue("ss", orig_str, \ dupe_str=reverse(strdup(orig_str))); free(dupe_str); return retval; } We introduce the dupe_str variable to point to the newly allocated string and build the return object. Then we free() the memory allocated and finally return back to the caller. Now we are done. Add PyMethodDef ModuleMethods[] Array/Table for Each Module FunctionNow that both of our wrappers are complete, we want to list them somewhere so that the Python interpreter knows how to import and access them. This is the job of theModuleMethods[] array. It is made up of an array of arrays, with each individual array containing information about each function, terminated by a NULL array marking the end of the list. For our Extest module, we create the following ExtestMethods[] array: static PyMethodDef ExtestMethods[] = { { "fac", Extest_fac, METH_VARARGS }, { "doppel", Extest_doppel, METH_VARARGS }, { NULL, NULL }, }; The Python-accessible names are given, followed by the corresponding wrapping functions. The constant METH_VARARGS is given, indicating a set of arguments in the form of a tuple. If we are using PyArg_ParseTupleAndKeywords() with keyworded arguments, we would logically OR this flag with the METH_KEYWORDS constant. Finally, a pair of NULLs properly terminates our list of two functions. Add void initModule() Module Initializer FunctionThe final piece to our puzzle is the module initializer function. This code is called when our module is imported for use by the interpreter. In this code, we make one call to Py_InitModule () along with the module name and the name of the ModuleMethods[] array so that the interpreter can access our module functions. For our Extest module, our initExtest() procedure looks like this: void initExtest() { Py_InitModule("Extest", ExtestMethods); } We are now done with all our wrapping. We add all this code to our original code from Extest1.c and merge the results into a new file called Extest2.c, concluding the development phase of our example. Another approach to creating an extension would be to make your wrapping code first, using "stubs" or test or dummy functions which will, during the course of development, be replaced by the fully functional pieces of implemented code. That way you can ensure that your interface between Python and C is correct, and then use Python to test your C code. 22.2.3. CompilationNow we are on to the compilation phase. In order to get your new wrapper Python extension to build, you need to get it to compile with the Python library. This task has been standardized (since 30) across platforms to make life a lot easier for extension writers. The distutils package is used to build, install, and distribute modules, extensions, and packages. It came about back in Python 2.0 and replaced the old 1.x way of building extensions using "makefiles." Using distutils, we can follow this easy recipe:
Create setup.pyThe next step is to create a setup.py file. The bulk of the work will be done by the setup() function. All the lines of code that come before that call are preparatory steps. For building extension modules, you need to create an Extension instance per extension. Since we only have one, we only need one Extension instance: Extension('Extest', sources=['Extest2.c']) The first argument is the (full) extension name, including any high-level packages if necessary. The name should be in full dotted-attribute notation. Ours is standalone, hence the name "Extest." sources is a list of all the source files. Again, we only have the one, Extest2.c. Now we are ready to call setup(). It takes a name argument for what it is building and a list of the items to build. Since we are creating an extension, we set it a list of extension modules to build as ext_modules. The syntax will be like this: setup('Extest', ext_modules=[...]) Since we only have one module, we combine the instantiation of our extension module into our call to setup(), setting the module name as "constant" MOD on the preceding line: MOD = 'Extest' setup(name=MOD, ext_modules=[ Extension(MOD, sources=['Extest2.c'])]) There are many more options to setup(), which are too numerous to list here. You can find out more about creating setup.py and calling setup() in the official Python documentation that we refer to at the end of this chapter. Example 22.2 shows the complete script that we are using for our example. Example 22.2. The Build Script (setup.py)
Compile and Link Your Code by Running setup.pyNow that we have our setup.py file, we can build our extension by running it with the "build" directive, as we have done here on our Mac (your output will differ based on the version of the operating system you are running as well as the version of Python you are using): $ python setup.py build running build running build_ext building 'Extest' extension creating build creating build/temp.macosx-10.x-fat-2.x gcc -fno-strict-aliasing -Wno-long-double -no-cpp- precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -I/usr/include -I/usr/local/include -I/sw/include -I/ usr/local/include/python2.x -c Extest2.c -o build/ temp.macosx-10.x-fat-2.x/Extest2.o creating build/lib.macosx-10.x-fat-2.x gcc -g -bundle -undefined dynamic_lookup -L/usr/lib -L/ usr/local/lib -L/sw/lib -I/usr/include -I/usr/local/ include -I/sw/include build/temp.macosx-10.x-fat-2.x/ Extest2.o -o build/lib.macosx-10.x-fat-2.x/Extest.so 22.2.4. Import and TestImport Your Module from PythonYour extension module will be created in the build/lib.* directory from where you ran your setup.py script. You can either change to that directory to test your module or install it into your Python distribution with: $ python setup.py install If you do install it, you will get the following output: running install running build running build_ext running install_lib copying build/lib.macosx-10.x-fat-2.x/Extest.so -> /usr/local/lib/python2.x/site-packages Now we can test out our module from the interpreter: >>> import Extest >>> Extest.fac(5) 120 >>> Extest.fac(9) 362880 >>> Extest.doppel('abcdefgh') ('abcdefgh', 'hgfedcba') >>> Extest.doppel("Madam, I'm Adam.") ("Madam, I'm Adam.", ".madA m'I, madaM") Test FunctionThe one last thing we want to do is to add a test function. In fact, we already have one, in the form of the main() function. Now, it is potentially dangerous to have a main() function in our code because there should only be one main() in the system. We remove this danger by changing the name of our main() to test() and wrapping it, adding Extest_test() and updating the ExtestMethods array so that they both look like this: static PyObject * Extest_test(PyObject *self, PyObject *args) { test(); return (PyObject*)Py_BuildValue(""); } static PyMethodDef ExtestMethods[] = { { "fac", Extest_fac, METH_VARARGS }, { "doppel", Extest_doppel, METH_VARARGS }, { "test", Extest_test, METH_VARARGS }, { NULL, NULL }, }; The Extest_test() module function just runs test() and returns an empty string, resulting in a Python value of None being returned to the caller. Now we can run the same test from Python: >>> Extest.test() 4! == 24 8! == 40320 12! == 479001600 reversing 'abcdef', we get 'fedcba' reversing 'madam', we get 'madam' >>> In Example 22.3, we present the final version of Extest2.c that was used to generate the output we just witnessed. Example 22.3. Python-Wrapped Version of C Library (Extest2.c)
In this example, we chose to segregate our C code from our Python code. It just kept things easier to read and is no problem with our short example. In practice, these source files tend to get large, and some choose to implement their wrappers completely in a different source file, i.e., ExtestWrappers.c or something of that nature. 22.2.5. Reference CountingYou may recall that Python uses reference counting as a means of keeping track of objects and deallocating objects no longer referenced as part of the garbage collection mechanism. When creating extensions, you must pay extra special attention to how you manipulate Python objects because you must be mindful of whether or not you need to change the reference count for such objects. There are two types of references you may have to an object, one of which is an owned reference, meaning that the reference count to the object is incremented by one to indicate your ownership. One place where you would definitely have an owned reference is where you create a Python object from scratch. When you are done with a Python object, you must dispose of your ownership, either by decrementing the reference count, transferring your ownership by passing it on, or storing the object. Failure to dispose of an owned reference creates a memory leak. You may also have a borrowed reference to an object. Somewhat lower on the responsibility ladder, this is where you are passed the reference of an object, but otherwise do not manipulate the data in any way. Nor do you have to worry about its reference count, as long as you do not hold on to this reference after its reference count has decreased to zero. You may convert your borrowed reference to an owned reference simply by incrementing an object's reference count. Python provides a pair of C macros which are used to change the reference count to a Python object. They are given in Table 22.3.
In our above Extest_test() function, we return None by building a PyObject with an empty string; however, it can also be accomplished by becoming an owner of the None object, PyNone, incrementing your reference count to it, and returning it explicitly, as in the following alternative piece of code: static PyObject * Extest_test(PyObject *self, PyObject *args) { test(); Py_INCREF(Py_None); return PyNone; } Py_INCREF() and Py_DECREF() also have versions that check for NULL objects. They are Py_XINCREF() and Py_XDECREF(), respectively. We strongly urge the reader to consult the Python documentation regarding extending and embedding Python for all the details with regard to reference counting (see the documentation reference in the Appendix). 22.2.6. Threading and the GILExtension writers must be aware that their code may be executed in a multithreaded Python environment. Back in Section 18.3.1, we introduced the Python Virtual Machine (PVM) and the Global Interpreter Lock (GIL) and described how only one thread of execution can be running at any given time in the PVM and that the GIL is responsible for keeping other threads from running. Furthermore, we indicated that code calling external functions such as in extension code would keep the GIL locked until the call returns. We also hinted that there was a remedy, a way for the extension programmer to release the GIL, for example before performing a system call. This is accomplished by "blocking" your code off to where threads may (and may not) run safely using another pair of C macros, Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. A block of code bounded by these macros will permit other threads to run. As with the reference counting macros, we urge you to consult with the documentation regarding extending and embedding Python as well as the Python/C API reference manual. |