Recipe9.12.Capturing the Output and Error Streams from a Unix Shell Command

Recipe 9.12. Capturing the Output and Error Streams from a Unix Shell Command

Credit: Brent Burley, Bradey Honsinger, Tobias Polzin, Jonathan Cano, Padraig Brady

Problem

You need to run an external process in a Unix-like environment and capture both the output and error streams from that external process.

Solution

The popen2 module lets you capture both streams, but you also need help from module fcntl, to make the streams nonblocking and thus avoid deadlocks, and from module select, to orchestrate the action:

import os, popen2, fcntl, select def makeNonBlocking(fd):     fl = fcntl.fcntl(fd, os.F_GETFL)     try:         fcntl.fcntl(fd, os.F_SETFL, fl | os.O_NDELAY)     except AttributeError:         fcntl.fcntl(fd, os.F_SETFL, fl | os.FNDELAY) def getCommandOutput(command):     child = popen2.Popen3(command, 1) # Capture stdout and stderr from command     child.tochild.close( )             # don't need to write to child's stdin     outfile = child.fromchild     outfd = outfile.fileno( )     errfile = child.childerr     errfd = errfile.fileno( )     makeNonBlocking(outfd)            # Don't deadlock! Make fd's nonblocking.     makeNonBlocking(errfd)     outdata, errdata = [  ], [  ]     outeof = erreof = False     while True:         to_check = [outfd]*(not outeof) + [errfd]*(not erreof)         ready = select.select(to_check, [  ], [  ]) # Wait for input         if outfd in ready[0]:             outchunk = outfile.read( )             if outchunk == '':                 outeof = True             else:                 outdata.append(outchunk)         if errfd in ready[0]:             errchunk = errfile.read( )             if errchunk == '':                 erreof = True             else:                 errdata.append(errchunk)         if outeof and erreof:             break         select.select([  ],[  ],[  ],.1) # Allow a little time for buffers to fill     err = child.wait( )     if err != 0:         raise RuntimeError, '%r failed with exit code %d\n%s' % (             command, err, ''.join(errdata))     return ''.join(outdata) def getCommandOutput2(command):     child = os.popen(command)     data = child.read( )     err = child.close( )     if err:         raise RuntimeError, '%r failed with exit code %d' % (command, err)

Discussion

This recipe shows how to execute a Unix shell command and capture the output and error streams in Python. By contrast, os.system sends both streams directly to the terminal. The function getCommandOutput presented in this recipe executes a command and returns the command's output. If the command fails, getCommandOutput raises an exception, using the text captured from the command's stderr as part of the exception's arguments.

Most of the complexity of this code is due to the difficulty of capturing both the output and error streams of the child process independently and at the same time. Normal (blocking) read calls may deadlock if the child is trying to write to one stream, and the parent is waiting for data on the other stream; so, the streams must be set to nonblocking, and select must be used to wait for data on either of the streams.

Note that the second select call is included just to add a 0.1-second sleep after each read. Counter intuitively, this allows the code to run much faster, since it gives the child time to put more data in the buffer. Without it, the parent may try to read only a few bytes at a time, which can be very expensive. Calling time.sleep(0.1) should be exactly equivalent, but since I was already, necessarily, calling select.select elsewhere in the recipe's code, I decided not to also import module time needlessly.

If you want to capture only the output and don't mind the error stream going to the terminal, you can use the much simpler code presented in getCommandOutput2. If you want to suppress the error stream altogether, that's easy, toojust append 2>/dev/null to the command. For example:

listing = getCommandOutput2('ls -1 2>/dev/null')

Another possibility is given by the os.popen4 function, which combines the output and error streams of the child process. However, in that case the streams are combined in a potentially messy way, depending on how they are buffered in the child process, so this recipe can help.

In Python 2.4, you can use class Popen, instead of popen2.Popen3, from the new standard library module subprocess. However, the issues highlighted in this recipe (namely, the need to use modules fcntl and select to make files nonblocking and coordinate the loop that interacts with the child process) aren't really affected by whether you use popen2 or subprocess.

This recipe does, as advertised, require a rather Unix-like underlying platform. Cygwin, which does a generally great job of emulating Unix on top of Windows, is not sufficient; for example, it offers no way to set files to nonblocking mode, nor to select on general files. (Under Windows, you are allowed to select only on sockets, not on other files.) If you must run on such problematic, non-Unix platforms, you may prefer a very different approach, based on using temporary files:

import os, tempfile def getCommandOutput(command):     outfile = tempfile.mktemp( )     errfile = tempfile.mktemp( )     cmd = "( %s ) > %s 2> %s" % (command, outfile, errfile)     err = os.system(cmd) >> 8     try:         if err != 0:             raise RuntimeError, '%r failed with exit code %d\n%s' % (                 command, err, file(errfile).read( ))         return file(outfile).read( )     finally:         os.remove(outfile)         os.remove(errfile)

Recipe9.12.Capturing the Output and Error Streams from a Unix Shell Command