Recipe15.7.Using Twisted Perspective Broker

Recipe 15.7. Using Twisted Perspective Broker

Credit: Simon Foster

Problem

You want to implement Python clients and servers for some distributed processing task, without repetitious "boilerplate" code, and with excellent performance and scalability characteristics.

Solution

Use the Perspective Broker (PB) subsystem of the Twisted framework. A PB server just subclasses the PB's Root class and adds remotely callable methods. Here is an example of a server script which adds just one remotely callable method, named Pong:

from twisted.spread import pb from twisted.internet import reactor PORT = 8992 class Ponger(pb.Root):     def remote_Pong(self, ball):         print 'CATCH', ball,         ball += 1         print 'THROW', ball         return ball reactor.listenTCP(PORT, pb.BrokerFactory(Ponger( ))) reactor.run( )

We could write an equally trivial script for the client side of the interaction, but let's instead have a rather feature-rich PB client, which deals with important issues often ignored in introductory examples of distributed programming, such as error handling:

from twisted.spread import pb from twisted.internet import reactor import sys PORT = 8992 DELAY         = 1 DOG_DELAY     = 2 RESTART_DELAY = 5 class Pinger(object):     def _ _init_ _(self, host):         self.ping = None         self.host = host         self.ball = 0         self._start( )     def _start(self):         print 'Waiting for Server', self.host         dfr = pb.getObjectAt(self.host, PORT, 30)         dfr.addCallbacks(self._gotRemote, self._remoteFail)     def _gotRemote(self, remote):         remote.notifyOnDisconnect(self._remoteFail)         self.remote = remote         self._ping( )     def _remoteFail(self, _ _):         if self.ping:             print 'ping failed, canceling and restarting'             self.ping.cancel( )             self.ping = None         self.restart = reactor.callLater(RESTART_DELAY, self._start)     def _watchdog(self):         print 'ping timed out, canceling and restarting'         self._start( )     def _ping(self):         self.dog = reactor.callLater(DOG_DELAY, self._watchdog)         self.ball += 1         print 'THROW', self.ball,         dfr = self.remote.callRemote('Pong', self.ball)         dfr.addCallbacks(self._pong, self._remoteFail)     def _pong(self, ball):         self.dog.cancel( )         print 'CATCH',  ball         self.ball = ball         self.ping = reactor.callLater(DELAY, self._ping) if _ _name_ _ == '_ _main_ _':     if len(sys.argv) != 2:         print 'Usage: %s serverhost' % sys.argv[0]         sys.exit(1)     host = sys.argv[1]     print 'Ping-pong client to host', host     Pinger(host)     reactor.run( )

Discussion

Twisted is a framework for asynchronous (also known as event-driven) programming of network clients, servers, proxies, and so on. The asynchronous programming model (which Twisted implements through the Reactor Design Pattern embodied in the twisted.internet.reactor module) provides excellent performance and scalability characteristics for Twisted-based programs.

Twisted also includes many subsystems that offer your programs ready-to-go networking functionality. One of these subsystems, Perspective Broker (PB), is implemented in the twisted.spread.pb module. PB lets you code distributed-programming clients and servers, with an ease of use that's most clearly displayed in the server program at the start of this recipe's Solution. In just a few lines of code, the server class is able to expose remotely callable methods: all it takes is subclassing the Root class of the pb module and naming each remotely callable method with a prefix of remote_.

Most of the client code in this recipe is concerned with diagnosing and handling possible problems and errors with the connection to the server. Specifically, if the connection fails for any reason, including a timeout diagnosed by the watchdog timer that the client sets up each time it pings, the client attempts to reconnect to the server. If you kill the server, the client keeps trying to reconnect, periodically, until you restart the server.

Error-handling apart, the client is essentially as simple as the server. In the method _start, the client calls function getObjectAt of module twisted.spread.pb, which takes as its arguments the server's host, a port number, and a "time-out" delay in seconds. As usual in Python networking, the host can be either a network name, such as localhost, or a string representing an IP address, such as 127.0.0.1.

If no problems arise, getObjectAt returns an object that proxies for the remote PB server. The proxy object, in turn, has a callRemote method, which takes as its arguments the method name as a string, followed by any arguments you are passing to the remote method. callRemote returns a Twisted deferred object, the lynchpin of Twisted's style of asynchronous (event-driven) programming. Learning to use deferreds effectively is the fundamental step in learning to program with Twisted.

A deferred object represents an event that may occur in the future (the success-case) or may end in failure. Given a deferred, you can add callbacks to it for both success and failure cases. (You can also chain callbacks, a possibility that this recipe does not exploit.) When the deferred's event occurs, Twisted calls your "success-case" callback, passing as its argument the "result" of the deferred. Alternatively, if the deferred ends in failure, Twisted calls your failure-case callback, passing as its argument a failure object that wraps a Python exception object.

As you see in this recipe, despite deferreds' potentially rich and vast functionality, their use is really quite simple in most cases. For example, in the failure cases, the client in this recipe wants to retry connecting: therefore, method _remoteFail accepts the failure-object argument with an argument name of "two underscores" (_ _), a common Python convention that indicates the argument will be ignored.

Recipe15.7.Using Twisted Perspective Broker