Mon and SNMP

Mon and SNMP "Proof of Concept" Recipe

SNMP is a protocol for network management, and a clustered environment of homogenous, or even semi-homogenous, hosts is the ideal way to take full advantage of it.

But before you can use SNMP, you'll need to install the Net-SNMP (formerly UCD-SNMP) package on each cluster node.

Step 1: Install Net-SNMP Client Software on Each Cluster Node

You can build and compile the net-snmp package using the source tar ball located in the chapter17 directory on the CD-ROM included with this book, but your distribution probably already contains the SNMP software. Red Hat, for example, uses the following two RPM packages:

• ucd-snmp

• ucd-snmp-utils

  Check to see whether you already have them with the command:

 #rpm -qa | grep ucd-snmp 

If you don't, you'll need to install them (see Appendix D).

Note 

We built the Net-SNMP software from source code earlier in this recipe because Mon requires the Perl SNMP support. However, this only needs to be done on the cluster node manager; it is not required on each cluster node (though there's no harm in compiling and installing SNMP from source code on all of your cluster nodes).

Step 2: Create the snmp.conf Configuration File

All cluster nodes use the same snmp.conf configuration file. You can create it by entering the following lines into the /etc/snmp/snmpd.conf file:

 syslocation "Room 133" [1] syscontact [e-mail] ← [2] rwcommunity private ← [3] rocommunity public ← [4] authtrapenable 1 ← [5] trapcommunity trapServ ← [6] trapsink localhost ← [7] trap2sink localhost ← [8] disk / 100000 ← [9] disk /var 100000 ← [9] 

[1]	→ Quoted string.
[2]	→ Who is responsible for this system?
[3]	→ Community names are really passwords.
[4]	→ Password for read-only information.
[5]	→ Report SNMP authentication failures.
[6]	→ Password to use when sending traps.
[7]	→ Where to send SNMP traps.
[8]	→ SNMP version 2 traps.
[9]	→ Disk partition and minimum freespace.

Note 

If you have an existing /etc/snmp/snmpd.conf file, you can make a backup copy and remove everything from the file except for the lines shown in this example.

The community read-write and read-only names in this file (rwcommunity and rocommunity) are really the SNMP "passwords" used to access and write to the SNMP MIBS for this agent. Do not use these common values unless you are building your system behind a firewall.

The last two disk lines tell SNMP to raise an alert when there is less than 100 MB of free disk space on either of the disk partitions / or /var.

Step 3: Start the SNMP Agent

The init script to start the SNMP agent should have been installed with the RPM package. Check for the existence of the file in the /etc/init.d directory, and then run the command:

 #service snmpd start 

or

 #/etc/init.d/snmpd start 

The /var/log/messages file should report:

 snmpd: snmpd startup succeeded 

Note 

Be sure snmpd starts each time the system boots on all of the cluster nodes. See "Respawning Services with init" in Chapter 1 for instructions on how to automatically restart the snmpd daemon on each cluster node if it fails.

Step 4: View the SNMP MIB Locally

Examine the MIB on a cluster node with the command:

 #snmpwalk localhost -v 1 -c public 

This command should produce a long report showing the MIB on the cluster node indicating that the local SNMP agent (snmpd) is responding to the query.

Now let's look at something useful with the command:

 #snmpwalk -v 1 localhost -c public .1.3.6.1.4.1.2021.9 

Note 

In this command, we are using a numeric value instead of a symbolic name to locate information in the SNMP management information base, or MIB. To find the symbolic or textual name of this numeric object identifier (OID) number, use:

     #snmptranslate .1.3.6.1.4.1.2021.9 

If you set the disk values as shown in the configuration file example, you should see a disk report like this:

 enterprises.ucdavis.dskTable.dskEntry.dskIndex.1 = 1 enterprises.ucdavis.dskTable.dskEntry.dskIndex.2 = 2 enterprises.ucdavis.dskTable.dskEntry.dskPath.1 = / enterprises.ucdavis.dskTable.dskEntry.dskPath.2 = /var enterprises.ucdavis.dskTable.dskEntry.dskDevice.1 = /dev/sdb2 enterprises.ucdavis.dskTable.dskEntry.dskDevice.2 = /dev/sda3 enterprises.ucdavis.dskTable.dskEntry.dskMinimum.1 = 100000 enterprises.ucdavis.dskTable.dskEntry.dskMinimum.2 = 100000 enterprises.ucdavis.dskTable.dskEntry.dskMinPercent.1 = -1 enterprises.ucdavis.dskTable.dskEntry.dskMinPercent.2 = -1 enterprises.ucdavis.dskTable.dskEntry.dskTotal.1 = 381121 enterprises.ucdavis.dskTable.dskEntry.dskTotal.2 = 253871 enterprises.ucdavis.dskTable.dskEntry.dskAvail.1 = 268888 enterprises.ucdavis.dskTable.dskEntry.dskAvail.2 = 162940 enterprises.ucdavis.dskTable.dskEntry.dskUsed.1 = 92554 enterprises.ucdavis.dskTable.dskEntry.dskUsed.2 = 77824 enterprises.ucdavis.dskTable.dskEntry.dskPercent.1 = 26 enterprises.ucdavis.dskTable.dskEntry.dskPercent.2 = 32 enterprises.ucdavis.dskTable.dskEntry.dskPercentNode.1 = 18 enterprises.ucdavis.dskTable.dskEntry.dskPercentNode.2 = 0 enterprises.ucdavis.dskTable.dskEntry.dskErrorFlag.1 = 0 enterprises.ucdavis.dskTable.dskEntry.dskErrorFlag.2 = 0 enterprises.ucdavis.dskTable.dskEntry.dskErrorMsg.1 = enterprises.ucdavis.dskTable.dskEntry.dskErrorMsg.2 = 

Note 

You see the University of California at Davis name in this disk space SNMP MIB because they developed it.

Notice especially the dskEntry.dskErrorFlag lines (shown in bold in this example). These lines indicate whether the disk is below (value 0) or above (value 1) the threshold you specified in the snmpd.conf file. Take a moment to modify the disk threshold value (in megabytes) to one that is less than the amount of free disk space on one of the partitions (use the command df -m to check). Then enter:

 #service snmpd restart 

or

 #/etc/init.d/snmpd restart 

Note 

A kill HUP to the PID of the snmpd daemon accomplishes the same thing.

Then enter the same snmpwalk command again:

 #snmpwalk -v 1 localhost -c public .1.3.6.1.4.1.2021.9 

You should now see that the error flag has been set to indicate that the disk partition is running out of disk space.

Step 5: Install the SNMP Monitoring Agents

We're now ready to tell Mon to examine the SNMP MIB on the cluster node manager. In this test, we are using Mon to monitor the SNMP agent running on the local host (the cluster node manager itself) as an example. You'll be able to do the same for the entire cluster by simply launching SNMP daemons on all of your cluster nodes and adding them to the Mon configuration file on the cluster node manager.

Note 

You must have installed the Perl SNMP package as described earlier in this chapter so that the SNMP monitoring scripts that come with the Mon package can run properly on the cluster node manager.

In this example, we will use the SNMP disk monitoring program called netsnmp-freespace.monitor located on the CD-ROM in the chapter17 directory and online at http://ftp.kernel.org/pub/software/admin/mon/contrib/index.html. To install, enter:

 #cp /mnt/CDROM/chapter17/netsnmp-freespace.monitor /usr/lib/mon/mon.d/ #chmod 755 /usr/lib/mon/mon.d/netsnmp-freespace.monitor #/usr/lib/mon/mon.d/netsnmp-freespace.monitor localhost 

If you downloaded this monitoring utility and you see the error message localhost returned an SNMP error: Unknown user name, modify the netsnmpfreespace.monitor script to force it to use Version 2 of the SNMP protocol. (See the netsnmp-freespace.monitor script included on the CD-ROM for this one-line change.)

If you need to use a community name other than public to be able to read the SNMP MIB (because you specified something other than public in the snmpd.conf file), use the -c option to the netsnmp-freespace.monitor script.

Examine the Output of netsnmp-freespace.monitor

This script returns nothing if a disk space threshold has not been violated. Try modifying the freespace threshold in /etc/snmp/snmpd.conf, and then restart the snmpd service (service snmpd restart or /etc/init.d/snmpd restart) and run this command again. When a disk is violating a freespace threshold, you'll see a report like this:

 #/usr/lib/mon/mon.d/netsnmp-freespace.monitor localhost localhost localhost:/(/dev/sdb2) total=372 used=91(26%) free=261 err=/: less than 300000 free (= 267767) 

Install Mon on all Cluster Nodes

You are now ready to put Mon to work and ask it to watch the disk space on all of your cluster nodes. To do so, install the /etc/snmp/snmpd.conf file you created on all of your cluster nodes, and start the snmpd daemon on each node.

You can use this sample mon.cf configuration file on the cluster node manager to add netsnmp-freespace.monitor.

 alertdir      = /usr/lib/mon/alert.d mondir        = /usr/lib/mon/mon.d logdir        = /usr/lib/mon/logs histlength    = 500 dtlogging     = yes dtlogfile     = /usr/lib/mon/logs/dtlog hostgroup clusternodes localhost watch clusternodes     service cluster-ping-check         interval 30s         monitor fping.monitor         period wd {Su-Sa}             alert mail.alert [e-mail]             upalert mail.alert [e-mail]             alertevery 1h     service disk-space-check         interval 15m         monitor netsnmp-freespace.monitor         period wd {Su-Sa}             alert mail.alert [e-mail]             upalert mail.alert [e-mail] 

For the moment, the clusternodes group contains only localhost (which should point to the loopback IP address 127.0.0.1 in the /etc/hosts file). Start Mon again, and you can generate alerts based on disk thresholds on the local system by modifying the /etc/snmp/snmpd.conf file and restarting the snmpd daemon.^[2]

Now that we have a proof of concept for an SNMP configuration using Mon and SNMP, let's build a configuration that can be used to monitor important system functions on each cluster node.

^[2]If you used the technique described in Chapter 1 to start snmpd from an entry in the /etc/inittab file, you need only enter the following command to restart snmpd:

 #killall snmpd