Monitoring Download Progress

One potential weakness in the examples presented so far in this chapter is that there hasn't been a way to monitor a download in progress. Sure, it's nice that a Deferred will pass you the results of a page once it's completely downloaded, but sometimes what you really need is to keep an eye on the download as it's happening.

3.5.1. How Do I Do That?

Again, the utility functions provided by twisted.web.client don't give you quite enough control. Define a subclass of client.HTTPDownloader, the factory class used for downloading a web page to a file. By overriding a couple of methods, you can keep track of a download in progress. The webdownload.py script in Example 3-6 shows how.

Example 3-6. webdownload.py

from twisted.web import client



class HTTPProgressDownloader(client.HTTPDownloader):



 def gotHeaders(self, headers):

 if self.status == '200': # page data is on the way

 if headers.has_key('content-length'):

 self.totalLength = int(headers['content-length'][0])

 else:

 self.totalLength = 0

 self.currentLength = 0.0

 print ''

 return client.HTTPDownloader.gotHeaders(self, headers)



 def pagePart(self, data):

 if self.status == '200':

 self.currentLength += len(data)

 if self.totalLength:

 percent = "%i%%" % (

 (self.currentLength/self.totalLength)*100)

 else:

 percent = '%dK' % (self.currentLength/1000)

 print "33[1FProgress: " + percent

 return client.HTTPDownloader.pagePart(self, data)



def downloadWithProgress(url, file, contextFactory=None, *args, **kwargs):

 scheme, host, port, path = client._parse(url)

 factory = HTTPProgressDownloader(url, file, *args, **kwargs)

 if scheme == 'https':

 from twisted.internet import ssl

 if contextFactory is None:

 contextFactory = ssl.ClientContextFactory( )

 reactor.connectSSL(host, port, factory, contextFactory)

 else:

 reactor.connectTCP(host, port, factory)

 return factory.deferred



if __name__ == "_ _main_ _":

 import sys

 from twisted.internet import reactor



 def downloadComplete(result):

 print "Download Complete."

 reactor.stop( )



 def downloadError(failure):

 print "Error:", failure.getErrorMessage( )

 reactor.stop( )



 url, outputFile = sys.argv[1:]

 downloadWithProgress(url, outputFile).addCallback(

 downloadComplete).addErrback(

 downloadError)

 reactor.run( )

Run webdownload.py with two arguments: the URL of a page to download and a filename in which to save it. As the command works, it will print updates on the download progress:

 $ python webdownload.py http://www.oreilly.com/ oreilly.html

 Progress: 100% <- updated during the download

 Download Complete.

If the web server doesn't return a Content-Length header indicating the total length of the download, it isn't possible to calculate the percentage complete. In this case, webdownload.py prints the number of kilobytes downloaded:

 $ python webdownload.py http://www.slashdot.org/ slashdot.html

 Progress: 60K <- updated during the download

 Download Complete.

 

3.5.2. How Does That Work?

HTTPProgressDownloader is a subclass of client.HTTPDownloader. It overrides the gotHeaders method to check for a Content-Length header that would indicate the total size of the page being downloaded. It also overrides the pagePart method, which is called each time a chunk of page data is received, to keep track of the number of bytes downloaded so far.

Each time a chunk of data comes in, HTTPProgressDownloader prints out a progress report. The string 33[1F is a terminal escape sequence that causes each line of the progress report to be written over the preceding line. This effect makes it look like the progress information is being updated in place.

The downloadWithProgress function contains code similar to that in Example 3-5 for parsing the requested URL, creating the HTTPProgressDownloader factory object, and initializing the connection. downloadComplete and downloadError are simple callback and errback handlers that print a message and stop the reactor.