Recipe2.5.Counting Lines in a File


Recipe 2.5. Counting Lines in a File

Credit: Luther Blissett

Problem

You need to compute the number of lines in a file.

Solution

The simplest approach for reasonably sized files is to read the file as a list of lines, so that the count of lines is the length of the list. If the file's path is in a string bound to a variable named thefilepath, all the code you need to implement this approach is:

count = len(open(thefilepath, 'rU').readlines( ))

For a truly huge file, however, this simple approach may be very slow or even fail to work. If you have to worry about humongous files, a loop on the file always works:

count = -1 for count, line in enumerate(open(thefilepath, 'rU')):     pass count += 1

A tricky alternative, potentially faster for truly humongous files, for when the line terminator is '\n' (or has '\n' as a substring, as happens on Windows):

count = 0 thefile = open(thefilepath, 'rb') while True:     buffer = thefile.read(8192*1024)     if not buffer:         break     count += buffer.count('\n') thefile.close( )

The 'rb' argument to open is necessary if you're after speedwithout that argument, this snippet might be very slow on Windows.

Discussion

When an external program counts a file's lines, such as wc -l on Unix-like platforms, you can of course choose to use that (e.g., via os.popen). However, it's generally simpler, faster, and more portable to do the line-counting in your own program. You can rely on almost all text files having a reasonable size, so that reading the whole file into memory at once is feasible. For all such normal files, the len of the result of readlines gives you the count of lines in the simplest way.

If the file is larger than available memory (say, a few hundred megabytes on a typical PC today), the simplest solution can become unacceptably slow, as the operating system struggles to fit the file's contents into virtual memory. It may even fail, when swap space is exhausted and virtual memory can't help any more. On a typical PC, with 256MB RAM and virtually unlimited disk space, you should still expect serious problems when you try to read into memory files above, say, 1 or 2 GB, depending on your operating system. (Some operating systems are much more fragile than others in handling virtual-memory issues under such overly stressed load conditions.) In this case, looping on the file object, as shown in this recipe's Solution, is better. The enumerate built-in keeps the line count without your code having to do it explicitly.

Counting line-termination characters while reading the file by bytes in reasonably sized chunks is the key idea in the third approach. It's probably the least immediately intuitive, and it's not perfectly cross-platform, but you might hope that it's fastest (e.g., when compared with recipe 8.2 in the Perl Cookbook).

However, in most cases, performance doesn't really matter all that much. When it does matter, the time-sink part of your program might not be what your intuition tells you it is, so you should never trust your intuition in this matterinstead, always benchmark and measure. For example, consider a typical Unix syslog file of middling size, a bit over 18 MB of text in 230,000 lines:

[situ@tioni nuc]$ wc nuc  231581 2312730 18508908 nuc

And consider the following testing-and-benchmark framework script, bench.py:

import time def timeo(fun, n=10):     start = time.clock( )     for i in xrange(n): fun( )     stend = time.clock( )     thetime = stend-start     return fun._ _name_ _, thetime import os def linecount_w( ):     return int(os.popen('wc -l nuc').read( ).split( )[0]) def linecount_1( ):     return len(open('nuc').readlines( )) def linecount_2( ):     count = -1     for count, line in enumerate(open('nuc')): pass     return count+1 def linecount_3( ):     count = 0     thefile = open('nuc', 'rb')     while True:         buffer = thefile.read(65536)         if not buffer: break         count += buffer.count('\n')     return count for f in linecount_w, linecount_1, linecount_2, linecount_3:     print f._ _name_ _, f( ) for f in linecount_1, linecount_2, linecount_3:     print "%s: %.2f"%timeo(f)

First, I print the line-counts obtained by all methods, thus ensuring that no anomaly or error has occurred (counting tasks are notoriously prone to off-by-one errors). Then, I run each alternative 10 times, under the control of the timing function timeo, and look at the results. Here they are, on the old but reliable machine I measured them on:

[situ@tioni nuc]$ python -O bench.py linecount_w 231581 linecount_1 231581 linecount_2 231581 linecount_3 231581 linecount_1: 4.84 linecount_2: 4.54 linecount_3: 5.02

As you can see, the performance differences hardly matter: your users will never even notice a difference of 10% or so in one auxiliary task. However, the fastest approach (for my particular circumstances, on an old but reliable PC running a popular Linux distribution, and for this specific benchmark) is the humble loop-on-every-line technique, while the slowest one is the fancy, ambitious technique that counts line terminators by chunks. In practice, unless I had to worry about files of many hundreds of megabytes, I'd always use the simplest approach (i.e., the first one presented in this recipe).

Measuring the exact performance of code snippets (rather than blindly using complicated approaches in the hope that they'll be faster) is very importantso important, indeed, that the Python Standard Library includes a module, timeit, specifically designed for such measurement tasks. I suggest you use timeit, rather than coding your own little benchmarks as I have done here. The benchmark I just showed you is one I've had around for years, since well before timeit appeared in the standard Python library, so I think I can be forgiven for not using timeit in this specific case!

See Also

The Library Reference and Python in a Nutshell sections on file objects, the enumerate built-in, os.popen, and the time and timeit modules; Perl Cookbook recipe 8.2.



Python Cookbook
Python Cookbook
ISBN: 0596007973
EAN: 2147483647
Year: 2004
Pages: 420

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net