Working with Asynchronous Results

After the reactor, Deferreds are probably the most important objects used in Twisted. You'll work with Deferreds frequently in Twisted applications, so it's essential to understand how they work. Deferreds can be a little confusing at first, but their purpose is simple: to keep track of an asynchronous action, and to get the result when the action is completed.

Deferreds can be illustrated this way: perhaps you've had the experience of going to one of those restaurants where, if there's going to be a wait for a table, you're given a little pager that will buzz when your table is ready. Having the pager is nice, because it gives you freedom to do other things while you're waiting for your table, instead of standing around the front of the restaurant feeling bored. You can take a walk outside, or even go next door and do some shopping. When a table (finally!) becomes available, the pager buzzes, and you head back inside the restaurant to be seated.

A Deferred is like that pager. It gives your program a way of finding out when some asynchronous task is finished, which frees it up to do other things in the meantime. When a function returns a Deferred object, it's saying that there's going to be some delay before the final result of the function is available. To control what happens when the result does become available, you can assign event handlers to the Deferred.

2.2.1. How Do I Do That?

When writing a function that starts an asynchronous action, return a Deferred object. When the action is competed, call the Deferred's callback method with the return value. If the action fails, call Deferred.errback with an exception. Example 2-4 shows a program that uses an asynchronous action to test connectivity to a given server and port.

When calling a function that returns a Deferred object, use the Deferred.addCallback method to assign a function to handle the results of the deferred action if it completes successfully. Use the Deferred.addErrback method to assign a function to handle the exception if the deferred action fails.

Example 2-4. connectiontest.py

from twisted.internet import reactor, defer, protocol



class CallbackAndDisconnectProtocol(protocol.Protocol):

 def connectionMade(self):

 self.factory.deferred.callback("Connected!")

 self.transport.loseConnection( )



class ConnectionTestFactory(protocol.ClientFactory):

 protocol = CallbackAndDisconnectProtocol



 def _ _init_ _(self):

 self.deferred = defer.Deferred( )



 def clientConnectionFailed(self, connector, reason):

 self.deferred.errback(reason)



def testConnect(host, port):

 testFactory = ConnectionTestFactory( )

 reactor.connectTCP(host, port, testFactory)

 return testFactory.deferred



def handleSuccess(result, port):

 print "Connected to port %i" % port

 reactor.stop( )



def handleFailure(failure, port):

 print "Error connecting to port %i: %s" % (

 port, failure.getErrorMessage( ))

 reactor.stop( )



if __name__ == "_ _main_ _":

 import sys

 if not len(sys.argv) == 3:

 print "Usage: connectiontest.py host port"

 sys.exit(1)



 host = sys.argv[1]

 port = int(sys.argv[2])

 connecting = testConnect(host, port)

 connecting.addCallback(handleSuccess, port)

 connecting.addErrback(handleFailure, port)

 reactor.run( )

Run this script from the command line with two arguments: the name of the server to connect to and the port to connect on. The output will look something like this:

 

 $ python connectiontest.py oreilly.com 80

 Connection to port 80.

Or, if you try to connect to a closed port:

 $ python connectiontest.py oreilly.com 81

 Error connecting to port 81: Connection was refused by other side: 22: Invalid

 argument.

Or if you try to connect to a server that doesn't really exist:

 $ python connectiontest.py fakesite 80

 Error connecting to port 80: DNS lookup failed: address 'fakesite' not found.

 

2.2.2. How Does That Work?

The ConnectionTestFactory is a ClientFactory that has a Deferred object as an attribute (called, predictably enough, deferred). When the connection is completed, the connectionMade method of the CallbackAndDisconnectProtocol will be called. connectionMade then calls self.factory.deferred.callback with an arbitrary value to indicate a successful result. If the connection fails, the ConnectionTestFactory's clientConnectionFailed method will be called. The second argument to clientConnectionFailed, reason, is a twisted.python.failure.Failure object encapsulating an exception describing why the connection failed. clientConnectionFailed passes the Failure to self.deferred.errback to indicate that the action failed.

The testConnect function takes two arguments: host and port. It creates a ConnectionTestFactory called testFactory, and passes it to reactor.connectTCP along with the host and port. It then returns a testFactory.deferred attribute, which is the Deferred object used to track whether the connection attempt succeeds.

Example 2-4 also defines two event handler functions: handleSuccess and handleFailure. When run from the command line, it takes the arguments for host and port and uses them to call testConnect, assigning the resulting Deferred to the variable connecting. It then uses connecting.addCallback and connecting.addErrback to set up the event handler functions. In each case, it passes port as an additional argument. Because any extra arguments or keyword arguments given to addCallback or addErrback will be passed on to the event handler, this has the effect of calling handleSuccess or handleFailure with the port as the second argument.

When your function returns a Deferred, you must make sure to eventually call either that Deferred's callback or errback method. Otherwise, the code that calls your function will be stuck waiting forever for a result.

After calling testConnect and setting up the event handlers, control is handed off to the reactor with a call to reactor.run( ). Depending on the success of testConnect, either handleSuccess or handleFailure will eventually be called, printing a description of what happened and stopping the reactor.

2.2.3. What About...

... keeping track of a bunch of related Deferreds? Sometimes you might want to write a program that does many asynchronous tasks at the same time, and then exits when they've all completed. For example, you might want to build a simple port scanner that runs the testConnect function from Example 2-4 against a range of ports. To do this, use the DeferredList object, as shown in Example 2-5, to manage a group of Deferreds.

Example 2-5. portscan.py

from twisted.internet import reactor, defer

from connectiontester import testConnect



def handleAllResults(results, ports):

 for port, resultInfo in zip(ports, results):

 success, result = resultInfo

 if success:

 print "Connected to port %i" % port

 reactor.stop( )



import sys

host = sys.argv[1]

ports = range(1, 201)

testers = [testConnect(host, port) for port in ports]

defer.DeferredList(testers, consumeErrors=True).addCallback(

 handleAllResults, ports)

reactor.run( )

Run portscan.py with a single argument: the host to scan. It will try to connect to that host on ports 1-200 and report on the results:

 $ python portscan.py  localhost 

 Connected to port 22

 Connected to port 23

 Connected to port 25

 Connected to port 80

 Connected to port 110

 Connected to port 139

 Connected to port 143

Example 2-5 uses a Python list comprehension to create a list of Deferred objects returned from testConnect( ). Each call to testConnect uses the host provided from the command line, and a port from 1 to 200:

 testers = [testConnect(host, port) for port in ports]

Then Example 2-5 wraps the list of Deferred objects in a DeferredList object. The DeferredList will track the results of all the Deferreds in the list passed in as its first argument. When they've all finished, it will call back with a list of tuples in the format (success, result) for each Deferred in the list. If the Deferred completed successfully, the first value in each tuple will be TRue and the second will contain the results returned by the Deferred; if it fails, the first value will be False and the second will contain a Failure object wrapping an exception. The consumeErrors keyword is set to TRue to tell the DeferredList to completely absorb any errors in its Deferreds. Otherwise, you'd see a bunch of messages about unhandled errors.

When the DeferredList is complete, the results are passed to the handleAllResults function, along with a list of which ports were scanned. handleAllResults uses the zip function to match each port with its results. For each port that had a successful connection, it prints a message. Finally, it stops the reactor to end the program.