Hack74.Profile Your Systems Using proc

Hack 74. Profile Your Systems Using /proc

The /proc filesystem holds a wealth of informationand with a little bit of scripting you can use it to create profiles of your servers.

The key to recognizing anomalies on your server is to have a good understanding and knowledge of what things look like when it's healthy. A great place to start hunting for information is the /proc filesystem. This filesystem is a portal into the depths of what the running kernel and the system load look like, and it provides a full profile of the hardware in use on the local system.

When I install a new server, one of the first things I do is take a sort of profile "snapshot," so that I can get a good picture of what the system resources look like on an idle system. I also do this just before and after I install or fire up new software or system services, so I can get a measure of an application's impact on the availability of system resources and so that I have a "cheat sheet" for looking up the system's installed hardware.

The script I use is very rough around the edges and wasn't written for the purpose of working on any machine you might ever run across, but it does work on a good number of Linux servers I've encountered. Let's have a look at each part of the script, along with the output it produces.

The first thing the script does is record the hostname and kernel version information, along with the first several lines of output from the top command, so I can see the load, number of users/processes, and so on:

 #!/bin/bash echo "" echo "#########BASIC SYSTEM INFORMATION########" echo HOSTNAME: `cat /proc/sys/kernel/hostname` echo DOMAIN: `cat /proc/sys/kernel/domainname` echo KERNEL: `uname -r` top -b | head -8 

Here's the output for this part of the script:

 #########BASIC SYSTEM INFORMATION######## HOSTNAME: willy DOMAIN: pvt KERNEL: 2.4.21-32.0.1.ELsmp  22:53:14 up 7 days, 15:36, 12 users, load average: 0.00, 0.02, 0.00 114 processes: 113 sleeping, 1 running, 0 zombie, 0 stopped CPU states: cpu  user nice  system  irq softirq  iowait  idle  total  0.0% 0.0%  0.4%   0.0% 0.0%  6.8%   92.6%  cpu00  0.0% 0.0%  0.9%   0.0% 0.0%  7.8% 91.1%  cpu01  0.0% 0.0%  0.0%   0.0% 0.0%     7.8% 92.1% 

The hostname information is there so I'll know what I'm looking at when I refer back to the output again in the future. The domain listed here is actually the NIS domain to which the box is bound. Depending on the environment, this can be an important bit of troubleshooting informationbut if you're in an NIS environment, you already knew that. What you're probably wondering is why I bothered to use /proc for this instead of system commands to get the hostname and domain name information. The answer is because I've found that using files under /proc is more reliable than assuming that system commands are in your default path. For things like hostname, chances are it's there, but three different tools can be installed for domain name information. A typical Red Hat host has domainname, ypdomainname, and dnsdomainname. On Red Hat systems, these are all symlinks to the hostname command. On my Debian stable box, there is no domainname command at all. However, the /proc/sys/kernel/domainname file is on most machines I come across, so using it makes the script more flexible.

Next up, let's have a look at the part of the script that gathers filesystem information:

 echo "######## FILESYSTEM INFORMATION #########" echo "" echo "SUPPORTED FILESYSTEM TYPES:" echo --------------------- echo `cat /proc/filesystems | awk -F'\t' '{print $2}'` echo "" echo "MOUNTED FILESYSTEMS:" echo --------------------- cat /proc/mounts 

Again, here's the output:

 SUPPORTED FILESYSTEM TYPES: -------------------------- sysfs rootfs bdev proc sockfs pipefs futexfs tmpfs eventpollfs devpts ext2 ramfs iso9660 devfs mqueue usbfs ext3 reiserfs supermount vfat MOUNTED FILESYSTEMS: ---------------------- /dev/root / reiserfs rw 0 0 none /dev devfs rw 0 0 none /proc proc rw,nodiratime 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 usbfs /proc/bus/usb usbfs rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/hdb1 /mnt/hdb1 ext3 rw,noatime 0 0 /dev/hdb2 /mnt/hdb2 reiserfs rw,noatime 0 0 

This is not information that's likely to change on a standalone server, but in a large environment with many NFS mounts and running automounters, it can be useful information to have. The supported filesystem information is also handy if you're in a shop that builds its own kernels, because it'll let you know if your new junior admin made the novice mistake of forgetting to add ext3 or vfat support to the kernel.

This next bit is only slightly more complex. It summarizes information about IDE devices, their model numbers, the devices they're assigned to on the system (hda, hdb, etc.), and, in case you don't recognize the models, exactly what kinds of devices they are. Here's the IDE device portion of the script:

 echo "IDE DEVICES BY CONTROLLER" echo ------------------------ for i in `ls /proc/ide | grep ide` do echo $i: for j in `ls /proc/ide/$i | grep hd` do echo "" echo " $j" echo " --------" echo " model: `cat /proc/ide/$i/$j/model`" echo " driver: `cat /proc/ide/$i/$j/driver`" echo " device type: `cat /proc/ide/$i/$j/media`" if [ -e /proc/ide/$i/$j/geometry ]; then echo " geometry:" `cat /proc/ide/$i/$j/geometry` fi echo "" done done 

And here's the output:

 ###### IDE SUBSYSTEM INFORMATION ######## IDE DEVICES BY CONTROLLER ------------------------- ide0: hdb -------- model: ST3200822A driver: ide-disk version 1.18 device type: disk ide1: hdd -------- model: FX4830T driver: ide-cdrom version 4.61 device type: cdrom 

This tells me that there are two IDE controllers, a CD-ROM drive, and one IDE hard drive on the machine. I also know that the CD drive is going to be mountable as /dev/hdd (something that might be less obvious on a machine with lots of IDE devices). Keep in mind that I could've gotten even more information if I wanted to require root privileges to run this script! For example, to see the settings for /dev/hdb, I need to be root. I can then run this command:

 # cat /proc/ide/hdb/settings 

This will give me more information than I could ever want to know about my hard drive. Here's a sampling:

 name value min max mode ---- ----- --- --- ---- acoustic 0 0 254 rw address 1 0 2 rw bios_cyl 24321 0 65535 rw bios_head 255 0 255 rw bios_sect 63 0 63 rw bswap 0 0 1 r current_speed 66 0 70 rw failures 0 0 65535 rw init_speed 66 0 70 rw io_32bit 1 0 3 rw keepsettings 0 0 1 rw lun 0 0 7 rw max_failures 1 0 65535 rw multcount 16 0 16 rw nice1 1 0 1 rw nowerr 0 0 1 rw number 1 0 3 rw pio_mode write-only 0 255 w unmaskirq 1 0 1 rw using_dma 1 0 1 rw wcache 1 0 1 rw 

There's a ton of information in the files under /proc. Scripts like this one can be greatly expanded upon and make a wonderful tool for consulting administrators. Send it to a client and have him send you the output via email, or use it to take a snapshot of a machine when you set it up so that when a client calls up you're ready with the information about the host in question.

8.7.1. The Code

Here's a copy of the entire script in one place, for easy review:

 #!/bin/bash echo "" echo "#########BASIC SYSTEM INFORMATION########" echo HOSTNAME: `cat /proc/sys/kernel/hostname` echo DOMAIN: `cat /proc/sys/kernel/domainname` echo KERNEL: `uname -r` top -b | head -8 echo "######## FILESYSTEM INFORMATION #########" echo "" echo "SUPPORTED FILESYSTEM TYPES:" cho --------------------- echo `cat /proc/filesystems | awk -F'\t' '{print $2}'` echo "" echo "MOUNTED FILESYSTEMS:" echo --------------------- cat /proc/mounts echo "IDE DEVICES BY CONTROLLER" echo ------------------------ for i in `ls /proc/ide | grep ide` do echo $i: for j in `ls /proc/ide/$i | grep hd` do echo "" echo " $j" echo " --------" echo " model: `cat /proc/ide/$i/$j/model`" echo " driver: `cat /proc/ide/$i/$j/driver`" echo " device type: `cat /proc/ide/$i/$j/media`" if [ -e /proc/ide/$i/$j/geometry ]; then  echo " geometry:" `cat /proc/ide/$i/$j/geometry` fi echo "" done done