Introduction

Credit: Jeremy Hylton, Google, Inc.

The recipes in this chapter describe simple techniques for using Python in distributed systems. Programming distributed systems is a difficult challenge, and recipes alone won't even come close to completely solving it. The recipes help you get programs on different computers talking to each other, so that you can start writing applications.

Remote Procedure Call (RPC) is an attractive approach to structuring a distributed system. The details of network communication are exposed through an interface that looks like normal procedure calls. When you call a function on a remote server, the RPC system is responsible for all the details of communication. It encodes the arguments so they can be passed over the network to the server, which might use different internal representations for the data. It invokes the right function on the remote machine and waits for a response.

The recipes in this chapter use three different systems that provide RPC interfacesCommon Object Request Broker Architecture (CORBA), Twisted's Perspective Broker (PB), and, in most recipes, XML-RPC. These systems are attractive because they make it easy to connect programs that can be running on different computers and might even be written in different languages. CORBA is a rather "heavyweight" protocol, very rich in functionality, with specialized and dedicated marshaling and transport layers (and much more besides). XML-RPC is a lightweight, simple-to-use protocol, which uses XML to marshal the call and its associated data, and HTTP for transport. Being simple and lightweight, XML-RPC is less functionally rich than CORBA. Both CORBA and XML-RPC are well-established standards, with implementations available for a wide variety of languages. In particular, XML-RPC is so simple and widespread that XML-RPC recipes take up half this chapter, a good, if rough, indication of how often Pythonistas are using it in preference to other distributed programming approaches.

PB is also "lightweight", while offering richer functionality than XML-RPC. However, PB is not a separate standard but a part of the Twisted framework, so PB implementations are currently limited to what Twisted itself provides and are mostly in Python. Perspective Broker is unusual among RPC systems because it openly exposes at application level the fact that network transactions are asynchronous, not synchronous like procedure calls within a single process. Therefore, in PB, the launching of a call to the remote procedure does not necessarily imply an immediate wait for the procedure's results; rather, the "result"s arrive "later" through a callback mechanism (specifically, Twisted's deferred objects). This asynchronous approach, which is the conceptual core of the Twisted framework, offers substantial advantages in terms of performance and scalability of the "result"ing network applications, but it may take some getting used to. Simon Foster's approach, shown in recipe, Recipe 15.7, is a simple way to get started exploring Perspective Broker.

XML-RPC is well supported by the Python Standard Library, with the xmlrpclib module for writing XML-RPC clients and the SimpleXMLRPCServer module for writing XML-RPC servers. For Twisted, CORBA, and other RPC standards yet (such as the emerging SOAPSimple Object Access Protocolsystem), you need to install third-party extensions before you can get started. The recipes in this chapter include pointers to the software you need. Unfortunately, you will not find pointers specifically to SOAP resources for Python in the recipes: for such pointers, I suggest you check out http://pywebsvcs.sourceforge.net/.

The Python Standard Library also provides a set of modules for doing the lower-level work of network programmingsocket, select, asyncore, and asynchat. The library also includes modules for marshaling data and sending it across sockets: struct, pickle, xdrlib. Chapter 13, includes recipes in which some of these modules are directly used, and Chapter 7, contains recipes dealing with serialization and marshaling. These lower-level modules, in turn, provide the plumbing for many other higher-level modules that directly support a variety of standard network protocols. Jeff Bauer offers Recipe 15.9, using the telnetlib module to send commands to remote machines via the Telnet protocol. Telnet is not a very secure protocol, and thus, except for use within well-protected LANs, has been largely supplanted by more secure protocols such as SSH (Secure Shell). Peter Cogolo and Anna Martelli Ravenscroft offer similar functionality to Bauer's, in Recipe 15.10, which uses SSH (via the third-party package paramiko) rather than Telnet.

Six of the recipes, just about half of the chapter, focus on XML-RPC. Rael Dornfest and Jeremy Hylton demonstrate how to write an XML-RPC client program that retrieves data from O'Reilly's Meerkat service. Recipe 15.1 is only three lines long (including the import statement): indeed, this extreme conciseness is the recipe's main appeal.

Brian Quinlan and Jeff Bauer contribute two different recipes for constructing XML-RPC servers. Quinlan, in Recipe 15.2, shows how to use the SimpleXMLRPCServer module from the Python Standard Library to handle incoming requests. Bauer's is Recipe 15.3. Medusa, like Twisted, is a framework for writing asynchronous network programs. In both cases, the libraries do most of the work; other than a few lines of initialization and registration, the server looks like normal Python code.

Christop Dietze (with contributions from Brian Quinlan and Jeff Bauer), in Recipe 15.4, elaborates on the XML-RPC server theme by showing how to add the ability that enables remote clients to terminate the server cleanly. Rune Hansen, in Recipe 15.5, shows how to add several minor but useful niceties to your XML-RPC servers.

Peter Arwanitis, in Recipe 15.6, demonstrates how to implement an XML-RPC server with Twisted and, at the same time, give your server a GUI, thanks to the wxPython GUI framework.

A strong alternative to XML-based protocols is CORBA, an object-based RPC mechanism using its own protocol, IIOP (Internet Inter-Orb Protocol). CORBA is a mature technology compared to XML-RPC (or, even more, SOAP, which isn't used in any of these recipesapparently, Pythonistas aren't doing all that much with SOAP yet). CORBA was introduced in 1991. The Python language binding was officially approved more recently, in February 2000, and several ORBs (Object Request Brokersroughly, CORBA servers) support Python. Duncan Grisby, a researcher at AT&T Laboratories in Cambridge (U.K.), describes the basics of getting a CORBA client and server running in Recipe 15.8, which uses omniORB, a free ORB, and the Python binding he wrote for it.

CORBA has a reputation for complexity, but Grisby's recipe makes it look straightforward. More steps are involved in the CORBA client than in the XML-RPC client example, but they are not difficult. To connect an XML-RPC client to a server, you just need a URL. To connect a CORBA client to a server, you need a URLa special corbaloc URLand you also need to know the server's interface. Of course, you need to know the interface regardless of protocol, but CORBA uses it explicitly. In general, CORBA offers more features than other distributed programming frameworksinterfaces, type checking, passing references to objects, and more. CORBA also supports just about every Python data type as argument or result.

Regardless of the protocols or systems you choose, the recipes in this chapter can help get you started. Inter-program communication is an important part of building a distributed system, but it's just one part. Once you have a client and server working, you'll find you have to deal with other interesting, challenging problemserror detection, concurrency, and security, to name a few. The recipes here won't solve those problems, but they will prevent you from getting caught up in unimportant details of the communication protocols. Rob Riggs in Recipe 15.11 presents a simple way to use HTTPS (as supported by the Python Standard Library module httplib) to authenticate SSL clients; Simon Foster's previously mentioned Perspective Broker recipe provides a way to implement one specific but frequent strategy for error detection and handling, namely periodically trying to reconnect to a server after a timeout or explicitly discovered network error.