Section 22.2. Integration Modes
22.2. Integration ModesThe last two technical chapters of this book introduce Python's tools for interfacing to the outside world and discuss both its ability to be used as an embedded language tool in other systems and its interfaces for extending Python scripts with new modules and types implemented in C-compatible languages. We'll also briefly explore other integration techniques that are less C specific, such as the Component Object Model (COM) and Jython. Our focus in these chapters is on tight integrationwhere control is transferred between languages by a simple, direct, and fast in-process function call. Although it is also possible to link components of an application less directly using Inter-Process Communication (IPC) and networking tools such as sockets and pipes that we explored earlier in the book, we are interested in this part of the book in more direct and efficient techniques. When you mix Python with components written in C (or other compiled languages) either Python or C can be "on top." Because of that, there are two distinct integration modes and two distinct APIs:
Extending generally has three main roles: to optimize programsrecoding parts of a program in C is a last-resort performance boost; to leverage existing librariesopening them up for use in Python code extends their reach; and to allow Python programs to do things not directly supported by the languagePython code cannot normally access devices at absolute memory addresses, for instance, but can call C functions that do. For example, the NumPy package for Python is largely an instance of extending at work: by integrating optimized numeric libraries, it turns Python into a flexible and efficient system that some compare to Matlab. Embedding typically takes the role of customizationby running user-configurable Python code, a system can be modified without shipping or building its full source code. For instance, some game systems provide a Python customization layer that can be used to modify the game or characters. Embedding is also sometimes used to route events to Python coded handlers. Python GUI toolkits, for example, usually employ embedding. Figure 22-1 sketches this traditional dual-mode integration model. In extending, control passes from Python through a glue layer on its way to C code. In embedding, C code processes Python objects and runs Python code by calling Python C API functions. In some models, things are not as clear-cut. For example, in the COM and CORBA systems, calls are passed through different kinds of intermediaries. Under cytpes, Python scripts make library calls rather than employing glue code. And in systems such as Pyrex, things are more different still. We will meet such alternative systems later in this part of the book. For now, our focus is on traditional Python/C integration models. Figure 22-1. Traditional integration model This chapter covers extending, and the next explores embedding. Although we will study these topics in isolation, keep in mind that many systems combine the two techniques. For instance, embedded Python code run from C can also import and call linked-in C extensions to interface with the enclosing application. And in callback-based systems, C code initially accessed through extending interfaces may later use embedding techniques to run Python callback handlers on events. For example, when we created buttons with Python's Tkinter GUI library earlier in the book, we called out to a C library through the extending API. When our GUI's user later clicked those buttons, the GUI C library caught the event and routed it to our Python functions with embedding. Although most of the details are hidden to Python code, control jumps often and freely between languages in such systems. Python has an open and reentrant architecture that lets you mix languages arbitrarily. 22.2.1. Presentation NotesBefore we get into details, I should also mention that Python/C integration is a big topic. In principle, the entire set of extern C functions in the Python system makes up its runtime interface. Because of that, these next two chapters focus on the tools commonly used to implement integration with external componentsjust enough to get you started. For additional examples beyond this book and its examples distribution, see the Python source code itself; its Modules and Objects directories are a wealth of code resources. Most of the Python built-ins we have used in this bookfrom simple things such as integers and strings to more advanced tools such as files, system calls, Tkinter, and the DBM files underlying shelves. Their utilization of integration APIs can be studied in Python's source code distribution as models for extensions of your own. In addition, Python's Extending and Embedding and Python/C API manuals are now reasonably complete, and they provide supplemental information to the presentation here. If you plan to do integration, you should browse these as a next step. For example, the manuals go into additional details about C extensions in threaded programs and multiple interpreters in embedded programs, which we will largely finesse here. These chapters also assume that you know basic C programming concepts. If you don't, you won't miss much by skipping or skimming these chapters. Typically, C developers code the extending and embedding interfaces of a system, and others do the bulk of the system's programming with Python alone. But if you know enough about C programming to recognize a need for an extension language, you probably already have the required background knowledge for this chapter. The good news in both chapters is that much of the complexity inherent in integrating Python with a static compiled language such as C can be automated with tools in the extension domain and higher-level APIs in the embedding world. For example, in this chapter we begin with a brief look at basic C API use, but then quickly move onto using the automated SWIG integration code generator in three examples and introduce alternatives such as ctypes, Pyrex, and Boost.Python at the end of the chapter. The next chapter takes a similar approach to embedding, with a basic introduction followed by a higher-level library. For now, let's get started with some extending fundamentals. |
