Creating Custom Metrics with gmetric

If you want to use Ganglia to report and graph a performance metric that is not included with the standard Ganglia distribution, you can write your own performance monitoring script to report to the locally running gmond on each cluster node with gmetric. You'll find sample gmetric scripts in the Gmetric Script Repository available online (http://ganglia.sourceforge.net/gmetric).

The following is a sample custom gmetric script to create four NFS client performance metrics.

 #!/bin/bash # # Linux NFS Client statistics # # Report number of NFS client read, write and getattr calls Because we were last called. # # (Use utility "nfsstat -c" to look at the same thing). # # Note: Uses temp files in /tmp # # GETATTR if [ -f /tmp/nfsclientgetattr ]; then         thisnfsgetattr=`cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\ n",$4}'`         lastnfsgetattr=`cat /tmp/nfsclientgetattr`         let "deltagetattr = thisnfsgetattr - lastnfsgetattr"         # echo "delta getattr $deltagetattr"         /usr/bin/gmetric -nnfsgetattr -v$deltagetattr -tuint16 -ucalls fi # READ if [ -f /tmp/nfsclientread ]; then         thisnfsread=`cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\ n",$9}'`         lastnfsread=`cat /tmp/nfsclientread`         let "deltaread = thisnfsread - lastnfsread"         # echo "delta read $deltaread"         /usr/bin/gmetric -nnfsread -v$deltaread -tuint16 -ucalls fi # WRITE if [ -f /tmp/nfsclientwrite ]; then         thisnfswrite=`cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\ n",$10} '`         lastnfswrite=`cat /tmp/nfsclientwrite`         let "deltawrite = thisnfswrite - lastnfswrite"         # echo "delta write $deltawrite"         /usr/bin/gmetric -nnfswrite -v$deltawrite -tuint16 -ucalls fi # NFS Quality Assurance RATIO (nfsqaratio) # If this value shrinks too much then perhaps an application # program change introduced excessive GETATTR calls into production. if [ "$deltagetattr" -ne 0 ];then          let "nfsqaratio = (deltaread + deltawrite) / deltagetattr"          /usr/bin/gmetric -nnfsqaratio -v$nfsqaratio -tuint16 -ucalls fi # Update the old values on disk for the next time around. (We ignore # the fact that they have probably already changed while we made this # calculation). cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\n",$9}'  > /tmp/ nfsclientread cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\n",$10}' > /tmp/ nfsclientwrite cat /proc/net/rpc/nfs | tail -1 | awk '{printf "%s\n",$4}'  > /tmp/ nfsclientgetattr 

Notice that this script calculates four custom NFS client performance metrics and then stores them in variables (deltagetattr, deltaread, deltawrite, nfsqaratio). It then reports these values using gmetric. The script divides the sum of the NFS read and write calls by the number of GETATTR calls to produce an NFS quality assurance ratio (nfsqaratio). As the value of the nfsqaratio gets bigger, there are fewer NFS GETATTR calls relative to the total NFS read and write calls—which is a good thing—but if this value shrinks while the system is busy doing numerous NFS read and write operations, it means that programs using the NFS filesystem are causing excessive NFS GETATTR calls. (As discussed in Chapter 16, an excessive number of GETATTR calls will cause your network filesystem to perform poorly.)

We'll see how to schedule this script as a batch job for execution on all cluster nodes in the next chapter.