Recipe 8.2. Measuring Memory Usage on LinuxCredit: Jean Brouwers ProblemYou need to monitor how much memory your Python application, running under Linux, is currently using. However, the standard library module resource does not work correctly on Linux. SolutionWe can code our own resource measurements based on Linux's /proc pseudo-filesystem: import os _proc_status = '/proc/%d/status' % os.getpid( ) _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0, 'KB': 1024.0, 'MB': 1024.0*1024.0} def _VmB(VmKey): ''' given a VmKey string, returns a number of bytes. ''' # get pseudo file /proc/<pid>/status try: t = open(_proc_status) v = t.read( ) t.close( ) except IOError: return 0.0 # non-Linux? # get VmKey line e.g. 'VmRSS: 9999 kB\n ...' i = v.index(VmKey) v = v[i:].split(None, 3) # split on runs of whitespace if len(v) < 3: return 0.0 # invalid format? # convert Vm value to bytes return float(v[1]) * _scale[v[2]] def memory(since=0.0): ''' Return virtual memory usage in bytes. ''' return _VmB('VmSize:') - since def resident(since=0.0): ''' Return resident memory usage in bytes. ''' return _VmB('VmRSS:') - since def stacksize(since=0.0): ''' Return stack size in bytes. ''' return _VmB('VmStk:') - since DiscussionEach of the functions in this recipe takes an optional argument since because the typical usage of these functions is to find out how much more memory (virtual, resident, or stack) has been used due to a certain section of code. Having since as an optional argument makes this typical usage quite simple and elegant: m0 = memory( ) section of code you're monitoring m1 = memory(m0) print 'The monitored section consumed', m1, 'bytes of virtual memory'. Getting and parsing the contents of pseudo-file /proc/pid/status is probably not the most efficient way to get data about memory usage, and it is not portable to non-Linux systems. However, it is a very simple and easy-to-code approach, and after all, on a non-Linux Unix system, you can use the resource module from the Python Standard Library. In fact, you can use resource on Linux, but the various fields relevant to memory consumption, such as ru_maxrss, all have a constant value of 0, just like the various memory-consumption fields in the output of the time shell command under Linux. The root cause of this situation is a limitation in the Linux implementation of the geTRusage system call, documented in man getrusage. See AlsoDocumentation on the resource standard library module in the Library Reference. |