Recipe8.2.Measuring Memory Usage on Linux


Recipe 8.2. Measuring Memory Usage on Linux

Credit: Jean Brouwers

Problem

You need to monitor how much memory your Python application, running under Linux, is currently using. However, the standard library module resource does not work correctly on Linux.

Solution

We can code our own resource measurements based on Linux's /proc pseudo-filesystem:

import os _proc_status = '/proc/%d/status' % os.getpid( ) _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,           'KB': 1024.0, 'MB': 1024.0*1024.0} def _VmB(VmKey):     ''' given a VmKey string, returns a number of bytes. '''     # get pseudo file  /proc/<pid>/status     try:         t = open(_proc_status)         v = t.read( )         t.close( )     except IOError:         return 0.0  # non-Linux?     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'     i = v.index(VmKey)     v = v[i:].split(None, 3)  # split on runs of whitespace     if len(v) < 3:         return 0.0  # invalid format?     # convert Vm value to bytes     return float(v[1]) * _scale[v[2]] def memory(since=0.0):     ''' Return virtual memory usage in bytes. '''     return _VmB('VmSize:') - since def resident(since=0.0):     ''' Return resident memory usage in bytes. '''     return _VmB('VmRSS:') - since def stacksize(since=0.0):     ''' Return stack size in bytes. '''     return _VmB('VmStk:') - since

Discussion

Each of the functions in this recipe takes an optional argument since because the typical usage of these functions is to find out how much more memory (virtual, resident, or stack) has been used due to a certain section of code. Having since as an optional argument makes this typical usage quite simple and elegant:

    m0 = memory( )     section of code you're monitoring     m1 = memory(m0)     print 'The monitored section consumed', m1, 'bytes of virtual memory'.

Getting and parsing the contents of pseudo-file /proc/pid/status is probably not the most efficient way to get data about memory usage, and it is not portable to non-Linux systems. However, it is a very simple and easy-to-code approach, and after all, on a non-Linux Unix system, you can use the resource module from the Python Standard Library.

In fact, you can use resource on Linux, but the various fields relevant to memory consumption, such as ru_maxrss, all have a constant value of 0, just like the various memory-consumption fields in the output of the time shell command under Linux. The root cause of this situation is a limitation in the Linux implementation of the geTRusage system call, documented in man getrusage.

See Also

Documentation on the resource standard library module in the Library Reference.



Python Cookbook
Python Cookbook
ISBN: 0596007973
EAN: 2147483647
Year: 2004
Pages: 420

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net