Section 7.6. Copying Directory Trees
7.6. Copying Directory TreesThe next three sections conclude this chapter by exploring a handful of additional utilities for processing directories (a.k.a. folders) on your computer with Python. They present directory copy, deletion, and comparison scripts that demonstrate system tools at work. All of these were born of necessity, are generally portable among all Python platforms, and illustrate Python development concepts along the way. Some of these scripts do something too unique for the visitor module's classes we've been applying in early sections of this chapter, and so require more custom solutions (e.g., we can't remove directories we intend to walk through). Most have platform-specific equivalents too (e.g., drag-and-drop copies), but the Python utilities shown here are portable, easily customized, callable from other scripts, and surprisingly fast. 7.6.1. A Python Tree Copy ScriptMy CD writer sometimes does weird things. In fact, copies of files with odd names can be totally botched on the CD, even though other files show up in one piece. That's not necessarily a showstopper; if just a few files are trashed in a big CD backup copy, I can always copy the offending files to floppies one at a time. Unfortunately, Windows drag-and-drop copies don't play nicely with such a CD: the copy operation stops and exits the moment the first bad file is encountered. You get only as many files as were copied up to the error, but no more. In fact, this is not limited to CD copies. I've run into similar problems when trying to back up my laptop's hard drive to another drivethe drag-and-drop copy stops with an error as soon as it reaches a file with a name that is too long to copy (common in saved web pages). The last 45 minutes spent copying is wasted time; frustrating, to say the least! There may be some magical Windows setting to work around this feature, but I gave up hunting for one as soon as I realized that it would be easier to code a copier in Python. The cpall.py script in Example 7-25 is one way to do it. With this script, I control what happens when bad files are foundI can skip over them with Python exception handlers, for instance. Moreover, this tool works with the same interface and effect on other platforms. It seems to me, at least, that a few minutes spent writing a portable and reusable Python script to meet a need is a better investment than looking for solutions that work on only one platform (if at all). Example 7-25. PP3E\System\Filetools\cpall.py
This script implements its own recursive tree traversal logic and keeps track of both the "from" and "to" directory paths as it goes. At every level, it copies over simple files, creates directories in the "to" path, and recurs into subdirectories with "from" and "to" paths extended by one level. There are other ways to code this task (e.g., other cpall variants in the book's examples distribution change the working directory along the way with os.chdir calls), but extending paths on descent works well in practice. Notice this script's reusable cpfile functionjust in case there are multigigabyte files in the tree to be copied, it uses a file's size to decide whether it should be read all at once or in chunks (remember, the file read method without arguments actually loads the entire file into an in-memory string). We choose fairly large file and block sizes, because the more we read at once in Python, the faster our scripts will typically run. This is more efficient than it may sound; strings left behind by prior reads will be garbage collected and reused as we go. Also note that this script creates the "to" directory if needed, but it assumes that the directory is empty when a copy starts up; be sure to remove the target directory before copying a new tree to its name (more on this in the next section). Here is a big book examples tree copy in action on Windows; pass in the name of the "from" and "to" directories to kick off the process, redirect the output to a file if there are too many error messages to read all at once (e.g., > output.txt), and run an rm shell command (or similar platform-specific tool) to delete the target directory first if needed: C:\temp>rm -rf cpexamples C:\temp>python %X%\system\filetools\cpall.py examples cpexamples Note: dirTo was created Copying... Copied 1356 files, 118 directories in 2.41999995708 seconds C:\temp>fc /B examples\System\Filetools\cpall.py cpexamples\System\Filetools\cpall.py Comparing files examples\System\Filetools\cpall.py and cpexamples\System\Filetools\cpall.py FC: no differences encountered At the time I wrote this example in 2000, this test run copied a tree of 1,356 files and 118 directories in 2.4 seconds on my 650 MHz Windows 98 laptop (the built-in time.time call can be used to query the system time in seconds). It runs a bit slower if some other programs are open on the machine, and may run arbitrarily faster or slower for you. Still, this is at least as fast as the best drag-and-drop I've timed on Windows. So how does this script work around bad files on a CD backup? The secret is that it catches and ignores file exceptions, and it keeps walking. To copy all the files that are good on a CD, I simply run a command line such as this one: C:\temp>python %X%\system\filetools\cpall_visitor.py g:\PP3rdEd\examples\PP3E cpexamples Because the CD is addressed as "G:" on my Windows machine, this is the command-line equivalent of drag-and-drop copying from an item in the CD's top-level folder, except that the Python script will recover from errors on the CD and get the rest. On copy errors, it prints a message to standard output and continues; for big copies, you'll probably want to redirect the script's output to a file for later inspection. In general, cpall can be passed any absolute directory path on your machine, even those that indicate devices such as CDs. To make this go on Linux, try a root directory such as /dev/cdrom or something similar to address your CD drive. 7.6.2. Recoding Copies with a Visitor-Based ClassWhen I first wrote the cpall script just discussed, I couldn't see a way that the visitor class hierarchy we met earlier would help. Two directories needed to be traversed in parallel (the original and the copy), and visitor is based on climbing one tree with os.path.walk. There seemed no easy way to keep track of where the script was in the copy directory. The trick I eventually stumbled onto is not to keep track at all. Instead, the script in Example 7-26 simply replaces the "from" directory path string with the "to" directory path string, at the front of all directory names and pathnames passed in from os.path.walk. The results of the string replacements are the paths to which the original files and directories are to be copied. Example 7-26. PP3E\System\Filetools\cpall_visitor.py
This version accomplishes roughly the same goal as the original, but it has made a few assumptions to keep code simple. The "to" directory is assumed not to exist initially, and exceptions are not ignored along the way. Here it is copying the book examples tree again on Windows: C:\temp>rm -rf cpexamples C:\temp>python %X%\system\filetools\cpall_visitor.py examples cpexamples -quiet Copying... Copied 1356 files, 119 directories in 2.09000003338 seconds C:\temp>fc /B examples\System\Filetools\cpall.py cpexamples\System\Filetools\cpall.py Comparing files examples\System\Filetools\cpall.py and cpexamples\System\Filetools\cpall.py FC: no differences encountered Despite the extra string slicing going on, this version runs just as fast as the original. For tracing purposes, this version also prints all the "from" and "to" copy paths during the traversal unless you pass in a third argument on the command line or set the script's verbose variable to False or 0: C:\temp>python %X%\system\filetools\cpall_visitor.py examples cpexamples Copying... d examples => cpexamples\ f examples\autoexec.bat => cpexamples\autoexec.bat f examples\cleanall.csh => cpexamples\cleanall.csh ...more deleted... d examples\System => cpexamples\System f examples\System\System.txt => cpexamples\System\System.txt f examples\System\more.py => cpexamples\System\more.py f examples\System\reader.py => cpexamples\System\reader.py ...more deleted... Copied 1356 files, 119 directories in 2.31000006199 seconds |
