Fixing DOS Filenames

The heart of the prior script was findFiles, a function than knows how to portably collect matching file and directory names in an entire tree, given a list of filename patterns. It doesn't do much more than the built-in find.find call, but can be augmented for our own purposes. Because this logic was bundled up in a function, though, it automatically becomes a reusable tool.

For example, the next script imports and applies findFiles, to collect all file names in a directory tree, by using the filename pattern * (it matches everything). I use this script to fix a legacy problem in the book's examples tree. The names of some files created under MS-DOS were made all uppercase; for example, spam.py became SPAM.PY somewhere along the way. Because case is significant both in Python and on some platforms, an import statement like "import spam" will sometimes fail for uppercase filenames.

To repair the damage everywhere in the thousand-file examples tree, I wrote and ran Example 5-6. It works like this: For every filename in the tree, it checks to see if the name is all uppercase, and asks the console user whether the file should be renamed with the os.rename call. To make this easy, it also comes up with a reasonable default for most new names -- the old one in all-lowercase form.

Example 5-6. PP2EPyToolsfixnames_all.py

#########################################################
# Use: "python ....PyToolsfixnames_all.py".
# find all files with all upper-case names at and below
# the current directory ('.'); for each, ask the user for
# a new name to rename the file to; used to catch old 
# uppercase file names created on MS-DOS (case matters on
# some platforms, when importing Python module files);
# caveats: this may fail on case-sensitive machines if 
# directory names are converted before their contents--the
# original dir name in the paths returned by find may no 
# longer exist; the allUpper heuristic also fails for 
# odd filenames that are all non-alphabetic (ex: '.');
#########################################################

import os, string
listonly = 0

def allUpper(name):
 for char in name:
 if char in string.lowercase: # any lowercase letter disqualifies
 return 0 # else all upper, digit, or special 
 return 1 

def convertOne(fname):
 fpath, oldfname = os.path.split(fname)
 if allUpper(oldfname):
 prompt = 'Convert dir=%s file=%s? (y|Y)' % (fpath, oldfname)
 if raw_input(prompt) in ['Y', 'y']:
 default = string.lower(oldfname)
 newfname = raw_input('Type new file name (enter=%s): ' % default)
 newfname = newfname or default
 newfpath = os.path.join(fpath, newfname)
 os.rename(fname, newfpath)
 print 'Renamed: ', fname
 print 'to: ', str(newfpath)
 raw_input('Press enter to continue')
 return 1
 return 0

if __name__ == '__main__':
 patts = "*" # inspect all file names
 from fixeoln_all import findFiles # reuse finder function
 matches = findFiles(patts)

 ccount = vcount = 0
 for matchlist in matches: # list of lists, one per pattern
 for fname in matchlist: # fnames are full directory paths
 print vcount+1, '=>', fname # includes names of directories 
 if not listonly: 
 ccount = ccount + convertOne(fname)
 vcount = vcount + 1
 print 'Converted %d files, visited %d' % (ccount, vcount)

As before, the findFiles function returns a list of simple filename lists, representing the expansion of all patterns passed in (here, just one result list, for the wildcard pattern * ).[5] For each file and directory name in the result, this script's convertOne function prompts for name changes; an os.path.split and an os.path.join call combination portably tacks the new filename onto the old directory name. Here is a renaming session in progress on Windows:

[5] Interestingly, using string '*' for the patterns list works the same as using list ['*'] here, only because a single-character string is a sequence that contains itself; compare the results of map(find.find, '*') with map(find.find, ['*']) interactively to verify.

C:	empexamples>python %X%PyToolsfixnames_all.py 
Using Python find
1 => ..cshrc
2 => .LaunchBrowser.out.txt
3 => .LaunchBrowser.py
...
 ...more deleted...
...
218 => .Ai
219 => .AiExpertSystem
220 => .AiExpertSystemTODO
Convert dir=.AiExpertSystem file=TODO? (y|Y)n 
221 => .AiExpertSystem\__init__.py
222 => .AiExpertSystemholmes
223 => .AiExpertSystemholmesREADME.1ST
Convert dir=.AiExpertSystemholmes file=README.1ST? (y|Y)y 
Type new file name (enter=readme.1st):
Renamed: .AiExpertSystemholmesREADME.1st
to: .AiExpertSystemholmes
eadme.1st
Press enter to continue
224 => .AiExpertSystemholmesREADME.2ND
Convert dir=.AiExpertSystemholmes file=README.2ND? (y|Y)y 
Type new file name (enter=readme.2nd): readme-more 
Renamed: .AiExpertSystemholmesREADME.2nd
to: .AiExpertSystemholmes
eadme-more
Press enter to continue
...
 ...more deleted...
...
1471 => .	odos.py
1472 => .	ounix.py
1473 => .xferall.linux.csh
Converted 2 files, visited 1473

This script could simply convert every all-uppercase name to an all-lowercase equivalent automatically, but that's potentially dangerous (some names might require mixed-case). Instead, it asks for input during the traversal, and shows the results of each renaming operation along the way.

5.3.1 Rewriting with os.path.walk

Notice, though, that the pattern-matching power of the find.find call goes completely unused in this script. Because it always must visit every file in the tree, the os.path.walk interface we studied in Chapter 2 would work just as well, and avoids any initial pause while a filename list is being collected (that pause is negligible here, but may be significant for larger trees). Example 5-7 is an equivalent version of this script that does its tree traversal with the walk callbacks-based model.

Example 5-7. PP2EPyToolsfixnames_all2.py

###############################################################
# Use: "python ....PyToolsfixnames_all2.py".
# same, but use the os.path.walk interface, not find.find;
# to make this work like the simple find version, puts of
# visiting directories until just before visiting their
# contents (find.find lists dir names before their contents);
# renaming dirs here can fail on case-sensitive platforms 
# too--walk keeps extending paths containing old dir names;
###############################################################

import os
listonly = 0
from fixnames_all import convertOne

def visitname(fname):
 global ccount, vcount
 print vcount+1, '=>', fname
 if not listonly:
 ccount = ccount + convertOne(fname)
 vcount = vcount + 1

def visitor(myData, directoryName, filesInDirectory): # called for each dir 
 visitname(directoryName) # do dir we're in now,
 for fname in filesInDirectory: # and non-dir files here
 fpath = os.path.join(directoryName, fname) # fnames have no dirpath
 if not os.path.isdir(fpath):
 visitname(fpath)
 
ccount = vcount = 0
os.path.walk('.', visitor, None)
print 'Converted %d files, visited %d' % (ccount, vcount)

This version does the same job, but visits one extra file (the topmost root directory), and may visit directories in a different order (os.listdir results are unordered). Both versions run in under a dozen seconds for the example directory tree on my computer.[6] We'll revisit this script, as well as the fixeoln line-end fixer, in the context of a general tree-walker class hierarchy later in this chapter.

[6] Very subtle thing: both versions of this script might fail on platforms where case matters, if they rename directoriesalong the way. If a directory is renamed before the contents of that directory have been visited (e.g., a directory SPAM renamed to spam), then later reference to the directory's contents using the old name (e.g., SPAM/filename) will no longer be valid on case-sensitive platforms. This can happen in the find.find version, because directories can and do show up in the result list before their contents. It's also a potential with the os.path.walk version, because the prior directory path (with original directory names) keeps being extended at each level of the tree. I only use this script on Windows (DOS), so I haven't been bitten by this in practice. Workarounds -- ordering find result lists, walking trees in a bottom-up fashion, making two distinct passes for files and directories, queuing up directory names on a list to be renamed later, or simply not renaming directories at all -- are all complex enough to be delegated to the realm of reader experiments. As a rule of thumb, changing a tree's names or structure while it is being walked is a risky venture.