Hack80.Keep a Constant Watch on Hosts

Hack 80. Keep a Constant Watch on Hosts

Monitor load or other statistics for multiple hosts on your desktop or on the command line.

rstatd is an RPC-based kernel statistics server that is either included with or available for every form of Unix I've ever used. It isn't something new. In fact, I suspect that its age might cause it to slip under the radar of younger admins, who might not know it if it hasn't appeared on the front page of Freshmeat recently. Hopefully, the information here will pique your interest in this very useful tool.

When I say that rstatd provides "kernel statistics," I'm referring to things such as CPU load, page swapping statistics, network IO statistics, and the like. Of course, providing this information to administrators in a way that is useful can sometimes be challenging, but there are a few tools available to help.

To make these tools useful, you must have a running rstatd daemon. Note that rstatd is dependent on the portmap daemon, which should already be running if you're using other RPC-based services such as NIS or NFS. To do a quick check to make sure these are running, you can run the following command:

 $ rpcinfo p program   vers   proto   port 100000 2    tcp   111 portmapper 100000 2    udp   111 portmapper 100001 3    udp   646 rstatd 100001     2      udp    646 rstatd 100001      1      udp    646 rstatd 

Without any other arguments, this will show you the status of the local host. If you put a hostname on the end of the above command, it will show you the status of a remote host. Now we're ready to point some tools at this host!

First and foremost among these tools is the standard rup command, which is available on Linux and other Unix platforms. It's a simple rstatd client utility, but with the right tools you can use it to produce output similar to that produced by the top commandonly instead of monitoring processes on the local host, you can monitor the load on multiple machines. Here's a command you can run to have a list of hosts, sorted by load average, updated every five seconds:

 $ watch -n 5 rup -l  host1 host2 host3 host4 host5  host3 up 12 days, 7:33, load average: 0.00, 0.00, 0.00 host4 up 12 days, 7:28, load average: 0.00, 0.00, 0.00 host1 up 12 days, 6:11, load average: 0.05, 0.04, 0.05 host2 up 12 days, 6:11, load average: 0.05, 0.04, 0.05 host5 up 12 days, 7:29, load average: 0.09, 0.06, 0.01 

This is okay if you have no access to any kind of graphical environment. Of course, it takes over your terminal, so you'll at least need to run it inside a screen session [Hack #34] or in a separate virtual terminal. Another problem here is that it's just simple raw data output; it doesn't alert you to any events, like host4's load going through the stratosphere.

For that, we can move into graphical clients. An old favorite of mine is xmeter, which was developed long ago and has since seemingly been forgotten and abandoned. Its configuration takes a little time to sift through (it's not graphical), but it does come with a manpage to help out, and once it's configured the only thing you'll ever have to change is the list of hosts to monitor. It provides configuration options to change the color of the output based on thresholds, so if the load of a machine gets to be a bit out of control, the color change is likely to catch your eye. Figure 9-2 shows a shot of xmeter monitoring the load on multiple hosts.

Figure 9-2. xmeter display monitoring load on multiple servers

A more recent development in the world of rstatd data-collection tools is jperfmeter, which is a Java-based, cross-platform monitor with a more polished interface and a graphical configuration tool. It does not yet (at the time of writing) support thresholds, and it's missing a few other finer details, but it's a brand new tool, so I'm sure it will get there at some point.

There are other tools available for remote server statistics monitoring, but you may also want to look into building your own, using either the Rstat::Client Perl module or the RPC or rstat interfaces for other languages, such as Python, Java, or C/C++.