After the reactor, Deferreds are probably the most important objects used in Twisted. You'll work with Deferreds frequently in Twisted applications, so it's essential to understand how they work. Deferreds can be a little confusing at first, but their purpose is simple: to keep track of an asynchronous action, and to get the result when the action is completed.
Deferreds can be illustrated this way: perhaps you've had the experience of going to one of those restaurants where, if there's going to be a wait for a table, you're given a little pager that will buzz when your table is ready. Having the pager is nice, because it gives you freedom to do other things while you're waiting for your table, instead of standing around the front of the restaurant feeling bored. You can take a walk outside, or even go next door and do some shopping. When a table (finally!) becomes available, the pager buzzes, and you head back inside the restaurant to be seated.
A Deferred is like that pager. It gives your program a way of finding out when some asynchronous task is finished, which frees it up to do other things in the meantime. When a function returns a Deferred object, it's saying that there's going to be some delay before the final result of the function is available. To control what happens when the result does become available, you can assign event handlers to the Deferred.
2.2.1. How Do I Do That?
When writing a function that starts an asynchronous action, return a Deferred object. When the action is competed, call the Deferred's callback method with the return value. If the action fails, call Deferred.errback with an exception. Example 2-4 shows a program that uses an asynchronous action to test connectivity to a given server and port.
When calling a function that returns a Deferred object, use the Deferred.addCallback method to assign a function to handle the results of the deferred action if it completes successfully. Use the Deferred.addErrback method to assign a function to handle the exception if the deferred action fails.
Example 2-4. connectiontest.py
from twisted.internet import reactor, defer, protocol class CallbackAndDisconnectProtocol(protocol.Protocol): def connectionMade(self): self.factory.deferred.callback("Connected!") self.transport.loseConnection( ) class ConnectionTestFactory(protocol.ClientFactory): protocol = CallbackAndDisconnectProtocol def _ _init_ _(self): self.deferred = defer.Deferred( ) def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) def testConnect(host, port): testFactory = ConnectionTestFactory( ) reactor.connectTCP(host, port, testFactory) return testFactory.deferred def handleSuccess(result, port): print "Connected to port %i" % port reactor.stop( ) def handleFailure(failure, port): print "Error connecting to port %i: %s" % ( port, failure.getErrorMessage( )) reactor.stop( ) if __name__ == "_ _main_ _": import sys if not len(sys.argv) == 3: print "Usage: connectiontest.py host port" sys.exit(1) host = sys.argv port = int(sys.argv) connecting = testConnect(host, port) connecting.addCallback(handleSuccess, port) connecting.addErrback(handleFailure, port) reactor.run( )
Run this script from the command line with two arguments: the name of the server to connect to and the port to connect on. The output will look something like this:
$ python connectiontest.py oreilly.com 80 Connection to port 80.
Or, if you try to connect to a closed port:
$ python connectiontest.py oreilly.com 81 Error connecting to port 81: Connection was refused by other side: 22: Invalid argument.
Or if you try to connect to a server that doesn't really exist:
$ python connectiontest.py fakesite 80 Error connecting to port 80: DNS lookup failed: address 'fakesite' not found.
2.2.2. How Does That Work?
The ConnectionTestFactory is a ClientFactory that has a Deferred object as an attribute (called, predictably enough, deferred). When the connection is completed, the connectionMade method of the CallbackAndDisconnectProtocol will be called. connectionMade then calls self.factory.deferred.callback with an arbitrary value to indicate a successful result. If the connection fails, the ConnectionTestFactory's clientConnectionFailed method will be called. The second argument to clientConnectionFailed, reason, is a twisted.python.failure.Failure object encapsulating an exception describing why the connection failed. clientConnectionFailed passes the Failure to self.deferred.errback to indicate that the action failed.
The testConnect function takes two arguments: host and port. It creates a ConnectionTestFactory called testFactory, and passes it to reactor.connectTCP along with the host and port. It then returns a testFactory.deferred attribute, which is the Deferred object used to track whether the connection attempt succeeds.
Example 2-4 also defines two event handler functions: handleSuccess and handleFailure. When run from the command line, it takes the arguments for host and port and uses them to call testConnect, assigning the resulting Deferred to the variable connecting. It then uses connecting.addCallback and connecting.addErrback to set up the event handler functions. In each case, it passes port as an additional argument. Because any extra arguments or keyword arguments given to addCallback or addErrback will be passed on to the event handler, this has the effect of calling handleSuccess or handleFailure with the port as the second argument.
After calling testConnect and setting up the event handlers, control is handed off to the reactor with a call to reactor.run( ). Depending on the success of testConnect, either handleSuccess or handleFailure will eventually be called, printing a description of what happened and stopping the reactor.
2.2.3. What About...
... keeping track of a bunch of related Deferreds? Sometimes you might want to write a program that does many asynchronous tasks at the same time, and then exits when they've all completed. For example, you might want to build a simple port scanner that runs the testConnect function from Example 2-4 against a range of ports. To do this, use the DeferredList object, as shown in Example 2-5, to manage a group of Deferreds.
Example 2-5. portscan.py
from twisted.internet import reactor, defer from connectiontester import testConnect def handleAllResults(results, ports): for port, resultInfo in zip(ports, results): success, result = resultInfo if success: print "Connected to port %i" % port reactor.stop( ) import sys host = sys.argv ports = range(1, 201) testers = [testConnect(host, port) for port in ports] defer.DeferredList(testers, consumeErrors=True).addCallback( handleAllResults, ports) reactor.run( )
Run portscan.py with a single argument: the host to scan. It will try to connect to that host on ports 1-200 and report on the results:
$ python portscan.py localhost Connected to port 22 Connected to port 23 Connected to port 25 Connected to port 80 Connected to port 110 Connected to port 139 Connected to port 143
Example 2-5 uses a Python list comprehension to create a list of Deferred objects returned from testConnect( ). Each call to testConnect uses the host provided from the command line, and a port from 1 to 200:
testers = [testConnect(host, port) for port in ports]
Then Example 2-5 wraps the list of Deferred objects in a DeferredList object. The DeferredList will track the results of all the Deferreds in the list passed in as its first argument. When they've all finished, it will call back with a list of tuples in the format (success, result) for each Deferred in the list. If the Deferred completed successfully, the first value in each tuple will be TRue and the second will contain the results returned by the Deferred; if it fails, the first value will be False and the second will contain a Failure object wrapping an exception. The consumeErrors keyword is set to TRue to tell the DeferredList to completely absorb any errors in its Deferreds. Otherwise, you'd see a bunch of messages about unhandled errors.
When the DeferredList is complete, the results are passed to the handleAllResults function, along with a list of which ports were scanned. handleAllResults uses the zip function to match each port with its results. For each port that had a successful connection, it prints a message. Finally, it stops the reactor to end the program.
Building Simple Clients and Servers
Web Services and RPC
NNTP Clients and Servers
Services, Processes, and Logging