12.4 Event Manager (EVM) Revisited

In a cluster, you will have multiple systems generating events. Prior to TruCluster Server version 5.0A, events generated in a cluster were only member-specific. In other words, every member would log its own events via syslog and/or binlog, and unless the administrator set up a central logging repository, the logs also resided in each member's /var/adm directory. Troubleshooting cluster problems was often a trying experience in a cluster prior to TruCluster Server version 5.0A because of the compartmentalization of the members' logs. For example, to troubleshoot a problem in an eight-member cluster, the administrator would need to look at the logs from eight systems. This could mean that there were up to nine logs per system (binary.errlog, messages, and the seven logs in /var/adm/syslog.dated) or seventy-two logs in an eight-member cluster. That's a lot of logs to merge, sort, search, and analyze! As we wrote in chapter 8, EVM was designed specifically to address this type of situation.

In a cluster, events can be logged to a particular member or to the entire cluster. There is an event attribute, cluster_event, that if set to true will generate that event clusterwide. Conversely, if it is set to false, then that event is generated on a per-member basis. The majority of the EVM event templates have the cluster_event attribute set to true. Furthermore, EVM has the ability to retrieve events from any or all members in the cluster.

12.4.1 What EVM Events Are Clusterwide?

To determine whether or not an event is clusterwide, you can use the evmwatch(8) command.

 # evmwatch -i | evmsort -A -s @cluster_event-:@name -t "@cluster_event%5 - @name" True    -    sys.unix.clu.caa.action_script True    -    sys.unix.clu.caa.app.error True    -    sys.unix.clu.caa.app.modified True    -    sys.unix.clu.caa.app.registered ... False   -    sys.unix False   -    sys.unix.binlog False   -    sys.unix.binlog.hw.backplane_raid_ctlr False   -    sys.unix.binlog.hw.console ...

The options for the evmwatch(8) and evmsort(8) commands can be found in their respective reference pages; however, here is an explanation of the options we used above.

evmwatch:

`-i`	– This option returns all loaded event templates.

We take the output and pipe it into the evmsort(8) command to sort the events first by the cluster_event and then by the name.

evmsort:

`-A`	– This option invokes the `evmshow` command to produce formatted output of the sorted events.
`-s`	– This option is the sort specification. In this case we are sorting the `cluster_name` in reverse alphabetical order, followed by the `name` in alphabetical order.
`-t`	– This option is used to define the "show template" which is set to display the `cluster_event` in a five-character (`%5`) field, followed by a space, a hyphen, a space, and the event `name`.

As of this writing, there are 338 events defined, and 142 of these events are clusterwide events.

 # evmwatch -A -i -t @cluster_event | wc -l        338

 # evmwatch -A -i -t @cluster_event | grep True | wc -l        142

12.4.2 How Can I Retrieve Events From a Member?

You can filter events by the member's hostname with the evmget(8), evmshow(8), and evmwatch commands and the "-f [host = hostname ]" option.

 # evmget -f "[host molari] AND [age < 1d]" | \ > evmshow -t " - @host_name: @name" -T "%T" 00:00:52 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:00:57 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:02:57 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:03:02 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:08:02 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:08:07 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:09:07 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 00:09:12 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 03:03:12 - molari.tcrhb.com: sys.unix.fs.advfs.fset.umount 03:03:12 - molari.tcrhb.com: sys.unix.clu.cfs.advfs.not_served 04:10:32 - molari.tcrhb.com: sys.unix.hw.net.niff.alert 04:10:37 - molari.tcrhb.com: sys.unix.hw.net.niff.alert

12.4.3 EVM Event Filters and Templates Revisited

When the TruCluster Server is installed, there are additional event templates added to the system specific to TruCluster Server. If you look in the /usr/share/evm/templates directory, you will discover a new clu subdirectory. Additionally, new cluster filter files are also included in the /usr/share/evm/filters directory. Figure 12-3 shows the added event filters and template files and their locations.

click to expand
Figure 12-3: EVM Additions in a Cluster

12.4.3.1 EVM Events Revisited

There are two cluster events on a standalone system.

 # evmwatch -i -f @sys:clu | evmsort -A -s @name -t @name sys.unix.clu.versw sys.unix.clu.versw.msg.success

On a cluster, however, there are forty-seven cluster events.

 # evmwatch -i -f @sys:clu | evmsort -A -s @name -t @name sys.unix.clu.caa.action_script sys.unix.clu.caa.app.error sys.unix.clu.caa.app.modified sys.unix.clu.caa.app.registered sys.unix.clu.caa.app.running sys.unix.clu.caa.app.stopped sys.unix.clu.caa.app.transition sys.unix.clu.caa.app.unregistered sys.unix.clu.caa.cli sys.unix.clu.caa.err sys.unix.clu.caa.res.modified sys.unix.clu.caa.res.registered sys.unix.clu.caa.res.unregistered sys.unix.clu.cfs.advfs.no_server sys.unix.clu.cfs.advfs.not_served sys.unix.clu.cfs.advfs.served sys.unix.clu.cfs.fs.no_server sys.unix.clu.cfs.fs.not_served sys.unix.clu.cfs.fs.served sys.unix.clu.clua.config sys.unix.clu.clua.leave sys.unix.clu.clua.netroute sys.unix.clu.clua.no_gated sys.unix.clu.clua.proxy sys.unix.clu.clua.routeport sys.unix.clu.clua.routerstart sys.unix.clu.clua.routerstop sys.unix.clu.cnx.member.join sys.unix.clu.cnx.member.leave sys.unix.clu.cnx.qdisk.loss sys.unix.clu.cnx.qdisk.regain sys.unix.clu.cnx.quorum.gain sys.unix.clu.cnx.quorum.loss sys.unix.clu.drd.new_accessnode sys.unix.clu.drd.no_accessnode sys.unix.clu.drd.server_add sys.unix.clu.drd.server_leave sys.unix.clu.member.add sys.unix.clu.member.delete sys.unix.clu.shutdown.abort sys.unix.clu.shutdown.caa_alert sys.unix.clu.shutdown.clean sys.unix.clu.shutdown.private sys.unix.clu.shutdown.start sys.unix.clu.versw sys.unix.clu.versw.msg.success sys.unix.clu.wall

12.4.3.2 EVM Filters Revisited

The filters that were added can be seen by using the evf(1) script that we created. We discussed the evf script in chapter 8.

 # evf -v -d /usr/share/evm/filters c[la][ua] /usr/share/evm/filters     caa.evf       @caa:caa              ->  [name sys.unix.clu.caa]       @caa:app              ->  [name sys.unix.clu.caa.app]       @caa:cli              ->  [name sys.unix.clu.caa.cli]       @caa:err              ->  [name sys.unix.clu.caa.err]       @caa:res              ->  [name sys.unix.clu.caa.res]     clu.evf       @clu:clu              -> [name sys.unix.clu]       @clu:cfs              -> [name sys.unix.clu.cfs]       @clu:cnx              -> [name sys.unix.clu.cnx]       @clu:drd              -> [name sys.unix.clu.drd]       @clu:member           -> [name sys.unix.clu.member]       @clu:shutdown         -> [name sys.unix.clu.shutdown]       @clu:wall             -> [name sys.unix.clu.wall]       @clu:caa              -> [name sys.unix.clu.caa]       @clu:clua             -> [name sys.unix.clu.clua]