23.6. A High-Level Embedding API: ppembed As you can probably tell from Example 23-14, embedded-mode integration code can very quickly become as complicated as extending code for nontrivial use. Today, no automation solution solves the embedding problem as well as SWIG addresses extending. Because embedding does not impose the kind of structure that extension modules and types provide, it's much more of an open-ended problem; what automates one embedding strategy might be completely useless in another. With a little upfront work, though, you can still automate common embedding tasks by wrapping up calls in higher-level APIs that make assumptions about common use cases. These APIs could handle things such as error detection, reference counts, data conversions, and so on. One such API, ppembed, is available in this book's examples distribution. It merely combines existing tools in Python's standard C API to provide a set of easier-to-use calls for running Python programs from C. 23.6.1. Running Objects with ppembed For instance, Example 23-15 demonstrates how to recode objects-err-low.c in Example 23-14, by linking ppembed's library files with your program. Example 23-15. PP3E\Integrate\Embed\ApiClients\object-api.c #include <stdio.h> #include "ppembed.h" main ( ) { /* with ppembed high-level api */ int failflag; PyObject *pinst; char *arg1="sir", *arg2="robin", *cstr; failflag = PP_Run_Function("module", "klass", "O", &pinst, "( )") || PP_Run_Method(pinst, "method", "s", &cstr, "(ss)", arg1, arg2); printf("%s\n", (!failflag) ? cstr : "Can't call objects"); Py_XDECREF(pinst); free(cstr); }
|
This file uses two ppembed calls (the names that start with PP) to make the class instance and call its method. Because ppembed handles error checks, reference counts, data conversions, and so on, there isn't much else to do here. When this program is run and linked with ppembed library code, it works like the original, but it is much easier to read, write, and debug: .../PP3E/Integrate/Embed/ApiClients$ objects-api brave sir robin
See the book's examples distribution for the makefile used to build this program; because it's similar to what we've seen and may vary widely on your system, we'll omit further build details in this chapter. 23.6.2. Running Code Strings with ppembed The ppembed API provides higher-level calls for most of the embedding techniques we've seen in this chapter. For example, the C program in Example 23-16 runs code strings to make the (now rarely used) string module capitalize a simple text. Example 23-16. PP3E\Integrate\Embed\ApiClients\codestring-low.c #include <Python.h> /* standard API defs */ void error(char *msg) { printf("%s\n", msg); exit(1); } main( ) { /* run strings with low-level calls */ char *cstr; PyObject *pstr, *pmod, *pdict; /* with error tests */ Py_Initialize( ); /* result = string.upper('spam') + '!' */ pmod = PyImport_ImportModule("string"); /* fetch module */ if (pmod == NULL) /* for namespace */ error("Can't import module"); pdict = PyModule_GetDict(pmod); /* string._ _dict_ _ */ Py_DECREF(pmod); if (pdict == NULL) error("Can't get module dict"); pstr = PyRun_String("upper('spam') + '!'", Py_eval_input, pdict, pdict); if (pstr == NULL) error("Error while running string"); /* convert result to C */ if (!PyArg_Parse(pstr, "s", &cstr)) error("Bad result type"); printf("%s\n", cstr); Py_DECREF(pstr); /* free exported objects, not pdict */ }
|
This C program file includes politically correct error tests after each API call. When run, it prints the result returned by running an uppercase conversion call in the namespace of the Python string module: .../PP3E/Integrate/Embed/ApiClients$ codestring-low SPAM!
You can implement such integrations by calling Python API functions directly, but you don't necessarily have to. With a higher-level embedding API such as ppembed, the task can be noticeably simpler, as shown in Example 23-17. Example 23-17. PP3E\Integrate\Embed\ApiClients\codestring-api.c #include "ppembed.h" #include <stdio.h> /* with ppembed high-level API */ main( ) { char *cstr; int err = PP_Run_Codestr( PP_EXPRESSION, /* expr or stmt? */ "upper('spam') + '!'", "string", /* code, module */ "s", &cstr); /* expr result */ printf("%s\n", (!err) ? cstr : "Can't run string"); /* and free(cstr) */ }
|
When linked with the ppembed library code, this version produces the same result as the former. Like most higher-level APIs, ppembed makes some usage mode assumptions that are not universally applicable; when they match the embedding task at hand, though, such wrapper calls can cut much clutter from programs that need to run embedded Python code. 23.6.3. Running Customizable Validations Our examples so far have been intentionally simple, but embedded Python code can do useful work as well. For instance, the C program in Example 23-18 calls ppembed functions to run a string of Python code fetched from a file that performs validation tests on inventory data. To save space, I'm not going to list all the components used by this example (you can find all of its source files and makefiles in the book's examples distribution package). Still, this file shows the embedding portions relevant to this chapter: it sets variables in the Python code's namespace to serve as input, runs the Python code, and then fetches names out of the code's namespace as results.[*] [*] This is more or less the kind of structure used when Python is embedded in HTML files in contexts such as the Active Scripting extension described in Chapter 18, except that the globals set here (e.g., PRODUCT) become names preset to web browser objects, and the code is extracted from a web page. It is not fetched from a text file with a known name.
Example 23-18. PP3E\Integrate\Embed\Inventory\order-string.c /* run embedded code-string validations */ #include <ppembed.h> #include <stdio.h> #include <string.h> #include "ordersfile.h" run_user_validation( ) { /* Python is initialized automatically */ int i, status, nbytes; /* caveat: should check status everywhere */ char script[4096]; /* caveat: should malloc a big-enough block */ char *errors, *warnings; FILE *file; file = fopen("validate1.py", "r"); /* customizable validations */ nbytes = fread(script, 1, 4096, file); /* load Python file text */ script[nbytes] = '\0'; status = PP_Make_Dummy_Module("orders"); /* application's own namespace */ for (i=0; i < numorders; i++) { /* like making a new dictionary */ printf("\n%d (%d, %d, '%s')\n", i, orders[i].product, orders[i].quantity, orders[i].buyer); PP_Set_Global("orders", "PRODUCT", "i", orders[i].product); /* int */ PP_Set_Global("orders", "QUANTITY", "i", orders[i].quantity); /* int */ PP_Set_Global("orders", "BUYER", "s", orders[i].buyer); /* str */ status = PP_Run_Codestr(PP_STATEMENT, script, "orders", "", NULL); if (status == -1) { printf("Python error during validation.\n"); PyErr_Print( ); /* show traceback */ continue; } PP_Get_Global("orders", "ERRORS", "s", &errors); /* can split */ PP_Get_Global("orders", "WARNINGS", "s", &warnings); /* on blanks */ printf("errors: %s\n", strlen(errors)? errors : "none"); printf("warnings: %s\n", strlen(warnings)? warnings : "none"); free(errors); free(warnings); PP_Run_Function("inventory", "print_files", "", NULL, "( )"); } } main(int argc, char **argv) /* C is on top, Python is embedded */ { /* but Python can use C extensions too */ run_user_validation( ); /* don't need sys.argv in embedded code */ }
|
There are a couple of things worth noticing here. First, in practice, this program might fetch the Python code file's name or path from configurable shell variables; here, it is loaded from the current directory. Second, you could also code this program by using straight API calls rather than ppembed, but each of the PP calls here would then grow into a chunk of more complex code. As coded, you can compile and link this file with Python and ppembed library files to build a program. The Python code run by the resulting C program lives in Example 23-19; it uses preset globals and is assumed to set globals to send result strings back to C. Example 23-19. PP3E\Integrate\Embed\Inventory\validate1.py # embedded validation code, run from C # input vars: PRODUCT, QUANTITY, BUYER # output vars: ERRORS, WARNINGS import string # all Python tools are available to embedded code import inventory # plus C extensions, Python modules, classes,.. msgs, errs = [], [] # warning, error message lists def validate_order( ): if PRODUCT not in inventory.skus( ): # this function could be imported errs.append('bad-product') # from a user-defined module too elif QUANTITY > inventory.stock(PRODUCT): errs.append('check-quantity') else: inventory.reduce(PRODUCT, QUANTITY) if inventory.stock(PRODUCT) / QUANTITY < 2: msgs.append('reorder-soon:' + repr(PRODUCT)) first, last = BUYER[0], BUYER[1:] # code is changeable onsite: if first not in string.uppercase: # this file is run as one long errs.append('buyer-name:' + first) # code string, with input and if BUYER not in inventory.buyers( ): # output vars used by the C app msgs.append('new-buyer-added') inventory.add_buyer(BUYER) validate_order( ) ERRORS = ' '.join(errs) # add a space between messages WARNINGS = ' '.join(msgs) # pass out as strings: "" == none
|
Don't sweat the details in this code; some components it uses are not listed here either (see the book's examples distribution for the full implementation). The thing you should notice, though, is that this code file can contain any kind of Python codeit can define functions and classes, use sockets and threads, and so on. When you embed Python, you get a full-featured extension language for free. Perhaps even more importantly, because this file is Python code, it can be changed arbitrarily without having to recompile the C program. Such flexibility is especially useful after a system has been shipped and installed. 23.6.3.1. Running function-based validations As discussed earlier, there are a variety of ways to structure embedded Python code. For instance, you can implement similar flexibility by delegating actions to Python functions fetched from module files, as illustrated in Example 23-20. Example 23-20. PP3E\Integrate\Embed\Inventory\order-func.c /* run embedded module-function validations */ #include <ppembed.h> #include <stdio.h> #include <string.h> #include "ordersfile.h" run_user_validation( ) { int i, status; /* should check status everywhere */ char *errors, *warnings; /* no file/string or namespace here */ PyObject *results; for (i=0; i < numorders; i++) { printf("\n%d (%d, %d, '%s')\n", i, orders[i].product, orders[i].quantity, orders[i].buyer); status = PP_Run_Function( /* validate2.validate(p,q,b) */ "validate2", "validate", "O", &results, "(iis)", orders[i].product, orders[i].quantity, orders[i].buyer); if (status == -1) { printf("Python error during validation.\n"); PyErr_Print( ); /* show traceback */ continue; } PyArg_Parse(results, "(ss)", &warnings, &errors); printf("errors: %s\n", strlen(errors)? errors : "none"); printf("warnings: %s\n", strlen(warnings)? warnings : "none"); Py_DECREF(results); /* ok to free strings */ PP_Run_Function("inventory", "print_files", "", NULL, "( )"); } } main(int argc, char **argv) { run_user_validation( ); }
|
The difference here is that the Python code file (shown in Example 23-21) is imported and so must live on the Python module search path. It also is assumed to contain functions, not a simple list of statements. Strings can live anywherefiles, databases, web pages, and so onand may be simpler for end users to code. But assuming that the extra requirements of module functions are not prohibitive, functions provide a natural communication model in the form of arguments and return values. Example 23-21. PP3E\Integrate\Embed\Inventory\validate2.py # embedded validation code, run from C # input = args, output = return value tuple import string import inventory def validate(product, quantity, buyer): # function called by name msgs, errs = [], [] # via mod/func name strings first, last = buyer[0], buyer[1:] if first not in string.uppercase: # or not first.isupper( ) errs.append('buyer-name:' + first) if buyer not in inventory.buyers( ): msgs.append('new-buyer-added') inventory.add_buyer(buyer) validate_order(product, quantity, errs, msgs) # mutable list args return ' '.join(msgs), ' '.join(errs) # use "(ss)" format def validate_order(product, quantity, errs, msgs): if product not in inventory.skus( ): errs.append('bad-product') elif quantity > inventory.stock(product): errs.append('check-quantity') else: inventory.reduce(product, quantity) if inventory.stock(product) / quantity < 2: msgs.append('reorder-soon:' + repr(product))
|
23.6.3.2. Other validation components For another API use case, the file order-bytecode.c in the book's source distribution shows how to utilize ppembed's convenience functions to precompile strings to bytecode for speed. It's similar to Example 23-18, but it calls PP_Compile_Codestr to compile and PP_Run_Bytecode to run. For reference, the database used by the validations code was initially prototyped for testing with the Python module inventory.py (see Example 23-22). Example 23-22. PP3E\Integrate\Embed\Inventory\inventory.py # simulate inventory/buyer databases while prototyping Inventory = { 111: 10, # "sku (product#) : quantity" 555: 1, # would usually be a file or shelve: 444: 100, # the operations below could work on 222: 5 } # an open shelve (or DBM file) too... Skus = Inventory.keys( ) # cache keys if they won't change def skus( ): return Skus def stock(sku): return Inventory[sku] def reduce(sku, qty): Inventory[sku] = Inventory[sku] - qty Buyers = ['GRossum', 'JOusterhout', 'LWall'] # or keys( ) of a shelve|DBM file def buyers( ): return Buyers def add_buyer(buyer): Buyers.append(buyer) def print_files( ): print Inventory, Buyers # check effect of updates
|
And the list of orders to process was simulated with the C header file ordersfile.h (see Example 23-23). Example 23-23. PP3E\Integrate\Embed\Inventory\ordersfile.h /* simulated file/dbase of orders to be filled */ struct { int product; /* or use a string if key is structured: */ int quantity; /* Python code can split it up as needed */ char *buyer; /* by convention, first-initial+last */ } orders[] = { {111, 2, "GRossum" }, /* this would usually be an orders file */ {222, 5, "LWall" }, /* which the Python code could read too */ {333, 3, "JOusterhout" }, {222, 1, "4Spam" }, {222, 0, "LTorvalds" }, /* the script might live in a database too */ {444, 9, "ERaymond" } }; int numorders = 6;
|
Both of these serve for prototyping, but are intended to be replaced with real database and file interfaces in later mutations of the system. See the WithDbase subdirectory in the book's source distribution for more on this thread. See also the Python-coded equivalents of the C files listed in this section; they were initially prototyped in Python too. And finally, here is the output produced the C string-based program in Example 23-18 when using the prototyping components listed in this section. The output is printed by C, but it reflects the results of the Python-coded validations it runs:[*] [*] Note that to get this example to work under Cygwin on Windows, I had to run the Python file through dos2unix to convert line-end characters; as always, your platform may vary.
.../PP3E/Integrate/Embed/Inventory$ ./order-string 0 (111, 2, 'GRossum') errors: none warnings: none {555: 1, 444: 100, 222: 5, 111: 8} ['GRossum', 'JOusterhout', 'LWall'] 1 (222, 5, 'LWall') errors: none warnings: reorder-soon:222 {555: 1, 444: 100, 222: 0, 111: 8} ['GRossum', 'JOusterhout', 'LWall'] 2 (333, 3, 'JOusterhout') errors: bad-product warnings: none {555: 1, 444: 100, 222: 0, 111: 8} ['GRossum', 'JOusterhout', 'LWall'] 3 (222, 1, '4Spam') errors: buyer-name:4 check-quantity warnings: new-buyer-added {555: 1, 444: 100, 222: 0, 111: 8} ['GRossum', 'JOusterhout', 'LWall', '4Spam'] 4 (222, 0, 'LTorvalds') Python error during validation. Traceback (most recent call last): File "<string>", line 25, in ? File "<string>", line 16, in validate_order ZeroDivisionError: integer division or modulo by zero 5 (444, 9, 'ERaymond') errors: none warnings: new-buyer-added {555: 1, 444: 91, 222: 0, 111: 8} ['GRossum', 'JOusterhout', 'LWall', '4Spam', 'LTorvalds', 'ERaymond']
The function-based output is similar, but more details are printed for the exception (function calls are active; they are not a single string). Trace through the Python and C code files to see how orders are validated and applied to inventory. This output is a bit cryptic because the system is still a work in progress at this stage. One of the nice features of Python, though, is that it enables such incremental development. In fact, with its integration interfaces, we can simulate future components in either Python or C. 23.6.4. ppembed Implementation The ppembed API originally appeared as an example in the first edition of this book. Since then, it has been utilized in real systems and has become too large to present here in its entirety. For instance, ppembed also supports debugging embedded code (by routing it to the pdb debugger module), dynamically reloading modules containing embedded code, and other features too complex to illustrate usefully here. But if you are interested in studying another example of Python embedding calls in action, ppembed's full source code and makefile live in this book's examples distribution: PP3E\Integration\Embed\HighLevelApi This API serves as a supplemental example of advanced embedding techniques. As a sample of the kinds of tools you can build to simplify embedding, the ppembed API's header file is shown in Example 23-24. You are invited to study, use, copy, and improve its code as you like. Or you can simply write an API of your own; the main point to take from this section is that embedding programs need to be complicated only if you stick with the Python runtime API as shipped. By adding convenience functions such as those in ppembed, embedding can be as simple as you make it. It also makes your C programs immune to changes in the Python C core; ideally, only the API must change if Python ever does. In fact, the third edition of this book proved this point: one of the utilities in the API had to be patched for a change in the Python/C API, but only one update was required. Be sure to also see the file abstract.h in the Python include directory if you are in the market for higher-level interfaces. That file provides generic type operation calls that make it easy to do things like creating, filling, indexing, slicing, and concatenating Python objects referenced by pointer from C. Also see the corresponding implementation file, abstract.c, as well as the Python built-in module and type implementations in the Python source distribution for more examples of lower-level object access. Once you have a Python object pointer in C, you can do all sorts of type-specific things to Python inputs and outputs. Example 23-24. PP3E\Integrate\Embed\HighLevelApi\ppembed.h /************************************************************************* * PPEMBED, VERSION 2.1 * AN ENHANCED PYTHON EMBEDDED-CALL INTERFACE * * Wraps Python's runtime embedding API functions for easy use. * Most utilities assume the call is qualified by an enclosing module * (namespace). The module can be a filename reference or a dummy module * created to provide a namespace for fileless strings. These routines * automate debugging, module (re)loading, input/output conversions, etc. * * Python is automatically initialized when the first API call occurs. * Input/output conversions use the standard Python conversion format * codes (described in the C API manual). Errors are flagged as either * a -1 int, or a NULL pointer result. Exported names use a PP_ prefix * to minimize clashes; names in the built-in Python API use Py prefixes * instead (alas, there is no "import" equivalent in C, just "from*"). * Also note that the varargs code here may not be portable to certain * C compilers; to do it portably, see the text or file 'vararg.txt' * here, or search for string STDARG in Python's source code files. * * New in version 2.1 (3rd Edition): minor fix for a change in the * Python C API: ppembed-callables.c call to _PyTuple_Resize -- added * code to manually move args to the right because the original * isSticky argument is now gone; * * New in version 2.0 (2nd Edition): names now have a PP_ prefix, * files renamed, compiles to a single file, fixed pdb retval bug * for strings, char* results returned by the "s" convert code now * point to new char arrays which the caller should free( ) when no * longer needed (this was a potential bug in prior version). Also * added new API interfaces for fetching exception info after errors, * precompiling code strings to byte code, and calling simple objects. * * Also fully supports Python 1.5 module package imports: module names * in this API can take the form "package.package.[...].module", where * Python maps the package names to a nested directories path in your * filesystem hierarchy; package dirs all contain _ _init_ _.py files, * and the leftmost one is in a directory found on PYTHONPATH. This * API's dynamic reload feature also works for modules in packages; * Python stores the full pathname in the sys.modules dictionary. * * Caveats: there is no support for advanced things like threading or * restricted execution mode here, but such things may be added with * extra Python API calls external to this API (see the Python/C API * manual for C-level threading calls; see modules rexec and bastion * in the library manual for restricted mode details). For threading, * you may also be able to get by with C threads and distinct Python * namespaces per Python code segments, or Python language threads * started by Python code run from C (see the Python thread module). * * Note that Python can only reload Python modules, not C extensions, * but it's okay to leave the dynamic reload flag on even if you might * access dynamically loaded C extension modules--in 1.5.2, Python * simply resets C extension modules to their initial attribute state * when reloaded, but doesn't actually reload the C extension file. *************************************************************************/ #ifndef PPEMBED_H #define PPEMBED_H #ifdef _ _cplusplus extern "C" { /* a C library, but callable from C++ */ #endif #include <stdio.h> #include <Python.h> extern int PP_RELOAD; /* 1=reload py modules when attributes referenced */ extern int PP_DEBUG; /* 1=start debugger when string/function/member run */ typedef enum { PP_EXPRESSION, /* which kind of code-string */ PP_STATEMENT /* expressions and statements differ */ } PPStringModes; /***************************************************/ /* ppembed-modules.c: load,access module objects */ /***************************************************/ extern char *PP_Init(char *modname); extern int PP_Make_Dummy_Module(char *modname); extern PyObject *PP_Load_Module(char *modname); extern PyObject *PP_Load_Attribute(char *modname, char *attrname); extern int PP_Run_Command_Line(char *prompt); /**********************************************************/ /* ppembed-globals.c: read,write module-level variables */ /**********************************************************/ extern int PP_Convert_Result(PyObject *presult, char *resFormat, void *resTarget); extern int PP_Get_Global(char *modname, char *varname, char *resfmt, void *cresult); extern int PP_Set_Global(char *modname, char *varname, char *valfmt, ... /*val*/); /***************************************************/ /* ppembed-strings.c: run strings of Python code */ /***************************************************/ extern int /* run C string of code */ PP_Run_Codestr(PPStringModes mode, /* code=expr or stmt? */ char *code, char *modname, /* codestr, modnamespace */ char *resfmt, void *cresult); /* result type, target */ extern PyObject* PP_Debug_Codestr(PPStringModes mode, /* run string in pdb */ char *codestring, PyObject *moddict); extern PyObject * PP_Compile_Codestr(PPStringModes mode, char *codestr); /* precompile to bytecode */ extern int PP_Run_Bytecode(PyObject *codeobj, /* run a bytecode object */ char *modname, char *resfmt, void *restarget); extern PyObject * /* run bytecode under pdb */ PP_Debug_Bytecode(PyObject *codeobject, PyObject *moddict); /*******************************************************/ /* ppembed-callables.c: call functions, classes, etc. */ /*******************************************************/ extern int /* mod.func(args) */ PP_Run_Function(char *modname, char *funcname, /* func|classname */ char *resfmt, void *cresult, /* result target */ char *argfmt, ... /* arg, arg... */ ); /* input arguments*/ extern PyObject* PP_Debug_Function(PyObject *func, PyObject *args); /* call func in pdb */ extern int PP_Run_Known_Callable(PyObject *object, /* func|class|method */ char *resfmt, void *restarget, /* skip module fetch */ char *argfmt, ... /* arg,.. */ ); /**************************************************************/ /* ppembed-attributes.c: run object methods, access members */ /**************************************************************/ extern int PP_Run_Method(PyObject *pobject, char *method, /* uses Debug_Function */ char *resfmt, void *cresult, /* output */ char *argfmt, ... /* arg, arg... */ ); /* inputs */ extern int PP_Get_Member(PyObject *pobject, char *attrname, char *resfmt, void *cresult); /* output */ extern int PP_Set_Member(PyObject *pobject, char *attrname, char *valfmt, ... /* val, val... */ ); /* input */ /**********************************************************/ /* ppembed-errors.c: get exception data after API error */ /**********************************************************/ extern void PP_Fetch_Error_Text( ); /* fetch (and clear) exception */ extern char PP_last_error_type[]; /* exception name text */ extern char PP_last_error_info[]; /* exception data text */ extern char PP_last_error_trace[]; /* exception traceback text */ extern PyObject *PP_last_traceback; /* saved exception traceback object */ #ifdef _ _cplusplus } #endif #endif /* !PPEMBED_H */
|
23.6.5. Other Integration Examples (External) While writing this chapter, I ran out of space before I ran out of examples. Besides the ppembed API example described in the last section, you can find a handful of additional Python/C integration self-study examples in this book's examples distribution:
PP3E\Integration\Embed\Inventory The full implementation of the validation examples listed earlier. This case study uses the ppembed API to run embedded Python order validations, both as embedded code strings and as functions fetched from modules. The inventory is implemented with and without shelves and pickle files for data persistence.
PP3E\Integration\Mixed\Exports A tool for exporting C variables for use in embedded Python programs.
PP3E\Integration\Embed\TestApi A ppembed test program, shown with and without package import paths to identify modules. Some of these are large C examples that are probably better studied than listed. |
