27.5 Modifying an Existing Package to Use EMS Resources

     

A generalized template to update a package can be summarized as follows :

1. If necessary, shut down the package(s).
   
   
2. If necessary, shut down the cluster.
   
   
3. Get the most up-to-date version of the package ASCII configuration file ( cmgetconf ).
   
   
4. Update the package ASCII configuration file.
   
   
5. Update and distribute the package control script, if necessary.
   
   
6. Update and distribute the application monitoring script(s), if necessary.
   
   
7. Check the package ASCII configuration file ( cmcheckconf ).
   
   
8. Compile and distribute the binary cluster configuration file ( cmapplyconf ).
   
   
9. Start the cluster, if necessary.
   
   
10. Start the package(s), if necessary.
    
    
11. Check the status of the package(s) and the cluster ( cmviewcl ).
    
    
12. Test the changes made.
    
    

We are looking at adding EMS resource monitoring to one of our packages. The Event Monitoring Service (EMS) is a system that allows machines to monitor the state and status of a number of local resources . EMS utilizes the SNMP protocol (via the snmpd daemon) whereby each machine maintains a list of applications or resources that it is monitoring; this list is known as a MIB (Management Information Base) structure. This structure is populated with many resources, anything from the status of hardware components to the available space in a filesystem to system performance metrics. EMS has a number of daemons associated with these different resources. As of Serviceguard version 10.10, a package can be made dependent on an MIB variable, i.e., if the state of a hardware or software component falls below a critical threshold, the application will deemed to have failed, whereby Serviceguard will move the package to an adoptive node. The use of EMS resources is becoming more popular as more hardware suppliers are updating EMS monitors for their components. If we review our cookbook for managing a cluster, we will see that adding an EMS resource to a package requires the cluster to be down. The reason for this is that the cmcld daemon needs to register its intent to monitor a resource via the relevant EMS daemons during its initialization.

The MIB structure utilized by the snmpd daemon is a tree-like structure similar to a UNIX filesystem. It has a root ( / ), and to navigate this structure, we use the command resls . Here are a couple of examples of traversing this structure using resls :

 root@hpeos001[] #  resls /  Contacting Registrar on hpeos001 NAME:   / DESCRIPTION:    This is the top level of the Resource Dictionary TYPE:   / is a Resource Class. There are 7 resources configured below /: Resource Class         /system         /StorageAreaNetwork         /adapters         /connectivity         /cluster         /storage         /net root@hpeos001[] #  resls /system  Contacting Registrar on hpeos001 NAME:   /system DESCRIPTION:    System Resources TYPE:   /system is a Resource Class. There are 8 resources configured below /system: Resource Class         /system/jobQueue1Min         /system/kernel_resource         /system/numUsers         /system/jobQueue5Min         /system/filesystem         /system/events         /system/jobQueue15Min         /system/status root@hpeos001[] # 

We look at configuring EMS in Chapter 10, "Monitoring System Resources." In that chapter, we see that EMS can be configured to send EMS traps to entities such as OpenView Network Node Manager management stations . In this instance, the trap will be relayed to the cmcld daemon, which will detect that the resource has breached a threshold and fail the associated package. If you spend some time traversing the MIB structure, you might come across some interesting resources that you could monitor which may relate to your applications. For example, if your application uses disk space under the /var filesystem, you could monitor the resource /system/filesystem/availMb :

 root@hpeos002[] #  resls /system/filesystem/availMb  Contacting Registrar on hpeos002 NAME:   /system/filesystem/availMb DESCRIPTION:    The amount of free space in the file system in megabytes. TYPE:   /system/filesystem/availMb is a Resource Class. There are 7 resources configured below /system/filesystem/availMb: Resource Class         /system/filesystem/availMb/home         /system/filesystem/availMb/opt         /system/filesystem/availMb/root         /system/filesystem/availMb/stand         /system/filesystem/availMb/tmp         /system/filesystem/availMb/usr         /system/filesystem/availMb/var root@hpeos002[] # 

Another valid resource could be the percentage of shared memory identifiers currently being used (kernel parameter = shmmni ). Many RDBMS applications use shared memory segments, and if we run out of shared memory identifiers, the application will be returned an ENOMEM error by the kernel. How it deals with this is application-dependent; it may cause the application to fail. It may be more acceptable to move the application to an adoptive node before we reach the time when there are no shared memory identifiers left; we can set a threshold that will trigger Serviceguard to move our package to an adoptive node.

Here's the output from resls for this resource:

 root@hpeos001[]  resls  /system/kernel_resource/system_v_ipc/shared_memory/shmmni  Contacting Registrar on hpeos001 NAME:   /system/kernel_resource/system_v_ipc/shared_memory/shmmni DESCRIPTION:    Percentage of shared memory ID consumption. TYPE:   /system/kernel_resource/system_v_ipc/shared_memory/shmmni is a Resource Instance         whose values are 64-bit floating point values. There are no active monitor requests reported for this resource. root@hpeos001[] 

We add this resource into one of our applications; it does not matter which one we choose, because we have to halt both of them in order to halt the cluster. I choose the clockwatch application to update. Looking at the template at the beginning of this section, I need to halt both the clockwatch and oracle1 packages before updating clockwatch to include my EMS resource. Here's the output from this process:

1. If necessary, shut down the package(s).
   
   

   
   

    root@hpeos002[] #  cmhaltpkg -v oracle1  Halting package oracle1. cmhaltpkg  : Successfully halted package oracle1. cmhaltpkg  : Completed successfully on all packages specified. root@hpeos002[] #  cmhaltpkg -v clockwatch  Halting package clockwatch. cmhaltpkg  : Successfully halted package clockwatch. cmhaltpkg  : Completed successfully on all packages specified. 

   
   
2. If necessary, shut down the cluster.
   
   

   
   

    root@hpeos002[] #  cmhaltcl -v  Disabling package switching to all nodes being halted. Disabling all packages from running on hpeos001. Disabling all packages from running on hpeos002. Disabling all packages from running on hpeos003. Warning:  Do not modify or enable packages until the halt operation is completed. This operation may take some time. Halting cluster services on node hpeos001. Halting cluster services on node hpeos003. Halting cluster services on node hpeos002. .. Successfully halted all nodes specified. Halt operation complete. root@hpeos002[] # 

   
   
3. Get the most up-to-date version of the package ASCII configuration file ( cmgetconf ).
   
   

   
   

    root@hpeos001[clockwatch] #  cmgetconf -v -p clockwatch clockwatch.curr  root@hpeos001[clockwatch] #  ll  total 244 -rwxr-x---   1 root       sys           1865 Aug  9 08:31 CLOCKWATCH.sh -rwxr-x---   1 root       sys          51467 Aug  9 08:32 clockwatch.cntl -rw-rw-rw-   1 root       root         35649 Aug  9 20:51 clockwatch.cntl.log -rwxr-x---   1 root       sys          12733 Aug  9 01:03 clockwatch.conf -rw-rw-rw-   1 root       sys          12701 Aug  9 20:56 clockwatch.curr root@hpeos001[clockwatch] # root@hpeos001[clockwatch] #  vi clockwatch.curr  

   
   
4. Update the package ASCII configuration file.
   
   Here are the lines I added at the end of the package ASCII configuration file for the resource in question:
   
   

   
   

    RESOURCE_NAME    /system/kernel_resource/system_v_ipc/shared_memory/s hmmni 

   
   This is the full pathname to the resource in question.
   
   

   
   

    RESOURCE_POLLING_INTERVAL  30 

   
   By default, EMS will monitor the resource every 60 seconds. Here, I am specifying a polling interval of every 30 seconds.
   
   

   
   

    RESOURCE_START             AUTOMATIC 

   
   If you are monitoring resources after the application is started up, e.g., filesystem space of shared filesystems, the RESOURCE_START needs to be set to DEFERRED . (This also requires a change to the application control script) In this case, we can start monitoring the resource before the application starts up.
   
   

   
   

    RESOURCE_UP_VALUE           < 75 

   
   The RESOURCE_UP_VALUE is setting the threshold for my resource. In this case, the resource is UP while its value is less than 75 percent. If 75 percent or more of the available shared memory identifiers get used up, this resource will be deemed DOWN, causing Serviceguard to move the package to an adoptive node. In this way, I am trying to preempt a failure by monitoring this critical system resource.
   
   
5. Update and distribute the package control script, if necessary.
   
   In this instance, this is not necessary.
   
   
6. Update and distribute the application monitoring script(s), if necessary.
   
   In this instance, this is not necessary.
   
   
7. Check the package ASCII configuration file (cmcheckconf).
   
   

   
   

    root@hpeos001[clockwatch] #  cmcheckconf -v -k -P clockwatch.curr  Checking existing configuration ... Done Gathering configuration information ... Done Parsing package file: clockwatch.curr. Package clockwatch already exists. It will be modified. Maximum configured packages parameter is 10. Configuring 2 package(s). 8 package(s) can be added to this cluster. Modifying the package configuration for package clockwatch. Adding resource ID 1 to package configuration. Verification completed with no errors found. Use the cmapplyconf command to apply the configuration. root@hpeos001[clockwatch] # 

   
   
8. Compile and distribute the binary cluster configuration file (cmapplyconf).
   
   

   
   

    root@hpeos001[clockwatch] #  cmapplyconf -v -k -P clockwatch.curr  Checking existing configuration ... Done Gathering configuration information ... Done Parsing package file: clockwatch.curr. Package clockwatch already exists. It will be modified. Maximum configured packages parameter is 10. Configuring 2 package(s). 8 package(s) can be added to this cluster. Modify the package configuration ([y]/n)?  y  Modifying the package configuration for package clockwatch. Adding resource ID 1 to package configuration. Completed the cluster update. root@hpeos001[clockwatch] # 

   
   
9. Start the cluster, if necessary.
   
   This is necessary in this case.
   
   

   
   

    root@hpeos001[clockwatch] #  cmruncl -v  Successfully started $SGLBIN/cmcld on hpeos003. Successfully started $SGLBIN/cmcld on hpeos001. Successfully started $SGLBIN/cmcld on hpeos002. cmruncl  : Waiting for cluster to form...... cmruncl  : Cluster successfully formed. cmruncl  : Check the syslog files on all nodes in the cluster cmruncl  : to verify that no warnings occurred during startup. root@hpeos001[clockwatch] # 

   
   
10. Start the package(s), if necessary.
    
    In our case, AUTO_RUN is enabled for the clockwatch package so it should start once the cluster is started.
    
    
11. Check the status of the package(s) and the cluster (cmviewcl).
    
    

    
    

     root@hpeos001[clockwatch] #  cmviewcl -v -p clockwatch  PACKAGE      STATUS       STATE        AUTO_RUN     NODE     clockwatch   up           running      enabled      hpeos001       Policy_Parameters:       POLICY_NAME     CONFIGURED_VALUE       Failover        configured_node       Failback        manual       Script_Parameters:       ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME       Service    up                  0         0   clock_mon       Subnet     up                                192.168.0.0       Resource   up                                /system/kernel_resource/system_v_ipc  /shared_memory/shmmni       Node_Switching_Parameters:       NODE_TYPE    STATUS       SWITCHING    NAME       Primary      up           enabled      hpeos001     (current)       Alternate    up           enabled      hpeos002       Alternate    up           enabled      hpeos003 root@hpeos001[clockwatch] # 

    
    
12. Test the changes made.
    
    With the resource we are looking at, we would have to wait until our application opened more and more shared memory segments. To breach 75 percent, my application would have to create +150 shared memory segments, e.g., 75 percent of the kernel parameter shmmni :
    
    

    
    

     root@hpeos001[clockwatch] #  kmtune -q shmmni  Parameter             Current Dyn Planned                  Module     Version ============================================================================= shmmni                    200  -  200 root@hpeos001[clockwatch] # 

    
    

As you can imagine, this may take some time. I decided to write a little program that spawns a child process that grabs and uses a 512KB shared memory segment (the program is called grabSHMEM and you can find the source code in Appendix B). This continues every 2 seconds until the resource is used up. Obviously, we wouldn't try this on a live production system, but it is of value to ensure that EMS is monitoring our resources properly. If you are going to try such a test, before you conduct the test please either:

1. Get the permission of a responsible adult.

2. Use a safety net.

The program grabSHMEM produces a line of output every time it is attached to a shared memory segment. If we kill the parent process (with signal INT , QUIT , or TERM ), all the children will detach and remove their shared memory segment. If you use any other signal, e.g., kill “9 on the parent, you will leave lots of shared memory segments still in use on your system; see ipcrm for details. Because it can take a few minutes to use up all the shared memory identifiers, I list only a section of the output from grabSHMEM :

 root@hpeos001[charlesk] #  ll grabSHMEM.c  -rw-rw-r--   1 root       sys           1547 Aug  9 23:05 grabSHMEM.c root@hpeos001[charlesk] #  make grabSHMEM  cc -O grabSHMEM.c  -o grabSHMEM (Bundled) cc: warning 480: The -O option is available only with the C/ANSI C product; ignored. root@hpeos001[charlesk] #  ./grabSHMEM &  [1]     3506 root@hpeos001[charlesk] # Attached to SHMEM ID = 31206 Attached to SHMEM ID = 807 Attached to SHMEM ID = 408 Attached to SHMEM ID = 409 Attached to SHMEM ID = 410 Attached to SHMEM ID = 411 

We can monitor the usage of shared memory segments with the command:

 root@hpeos001[] #  ipcs -mbop  IPC status from /dev/kmem as of Sun Aug 10 03:04:29 2003 T      ID     KEY        MODE      OWNER     GROUP NATTCH  SEGSZ  CPID  LPID Shared Memory: m       0 0x411c05c4 --rw-rw-rw-    root      root      0    348   420   420 m       1 0x4e0c0002 --rw-rw-rw-    root      root      1  61760   420   420 m       2 0x41200888 --rw-rw-rw-    root      root      1   8192   420   432 m       3 0x301c3e60 --rw-rw-rw-    root      root      3 1048576   957   971 m       4 0x501c7c07 --r--r--r--    root     other      1 1075095  1023  1023 m       5 0x00000000 --rw-------    root      root      2  65536  1308  1308 m   31206 0x00000000 --rw-rw-rw-    root       sys      1 524288  3507  3507 m     807 0x00000000 --rw-rw-rw-    root       sys      1 524288  3513  3513 m     408 0x00000000 --rw-rw-rw-    root       sys      1 524288  3514  3514 m     409 0x00000000 --rw-rw-rw-    root       sys      1 524288  3515  3515 m     410 0x00000000 --rw-rw-rw-    root       sys      1 524288  3516  3516 m     411 0x00000000 --rw-rw-rw-    root       sys      1 524288  3517  3517 

We can monitor the number of shared memory segments in use by using the command:

 root@hpeos001[] #  ipcs -mbop  tail +4  wc -l  136 

As you can see, we are nearly at a point where the EMS resource will reach its DOWN state (it took a few minutes to get to this point). We should then see the clockwatch package move to its adoptive node, in this case, hpeos002 . We can monitor this via cmviewcl and also syslog.log . Here is some of the output I received in syslog.log :

 Aug 10 03:15:55 hpeos001 cmcld: Resource /system/kernel_resource/system_v_ipc  /shared_memory/shmmni in package clockwatch does not meet RESOURCE_UP_VALUE. Aug 10 03:15:55 hpeos001 cmcld: Executing '/etc/cmcluster/clockwatch/clockwatch.cntl  stop' for package clockwatch, as service PKG*62977. Aug 10 03:16:00 hpeos001 CM-clockwatch[4198]: cmhaltserv clock_mon Aug 10 03:16:02 hpeos001 CM-clockwatch[4205]: cmmodnet -r -i 192.168.0.220 192.168.0.0 Aug 10 03:16:02 hpeos001 cmcld: Service PKG*62977 terminated due to an exit(0). Aug 10 03:16:02 hpeos001 cmcld: Halted package clockwatch on node hpeos001. Aug 10 03:16:02 hpeos001 cmcld: Resource /system/kernel_resource/system_v_ipc/sh ared_memory/shmmni does not meet package RESOURCE_UP_VALUE for package clockwatch. Aug 10 03:16:02 hpeos001 cmcld: Package clockwatch cannot run on this node. Aug 10 03:16:07 hpeos001 cmcld: (hpeos002) Started package clockwatch on node hpeos002. 

As you can see, node hpeos001 did not have the resources to continue running this application. As a result clockwatch was moved to its adoptive node ( hpeos002 ). Here's the output from cmviewcl :

 root@hpeos001[charlesk] #  cmviewcl -v -p clockwatch  PACKAGE      STATUS       STATE        AUTO_RUN     NODE     clockwatch   up           running      enabled      hpeos002       Policy_Parameters:       POLICY_NAME     CONFIGURED_VALUE       Failover        configured_node       Failback        manual       Script_Parameters:       ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME       Service    up                  0         0   clock_mon       Subnet     up                                192.168.0.0       Resource   up                                /system/kernel_resource/system_v_ipc  /shared_memory/shmmni       Node_Switching_Parameters:       NODE_TYPE    STATUS       SWITCHING    NAME       Primary      up           enabled      hpeos001       Alternate    up           enabled      hpeos002     (current)       Alternate    up           enabled      hpeos003 root@hpeos001[charlesk] # 

Notice that node hpeos001 is still enabled to run the application even though the EMS resource effectively failed the application. I would want to investigate this further and determine what has happened to all my shared memory segments. In this test scenario, my program grabSHMEM is still running and has consumed all the available shared memory segments:

 root@hpeos001[clockwatch] #  ipcs -mbop  tail +4  wc -l  200 

I am also getting errors from the program itself:

 NO shmem left: No space left on device 

When I killed the parent background process, I got lots of messages of the form:

 Received signal 15 Letting go of shmseg = 448 Received signal 15 Letting go of shmseg = 61 Received signal 15 Letting go of shmseg = 62 Received signal 15 Letting go of shmseg = 627 Received signal 15 Letting go of shmseg = 231 Received signal 15 Letting go of shmseg = 233 

This tells me that the shared memory segments have been released back to the system. I checked this with the following command:

 root@hpeos001[clockwatch] #  ipcs mbop  IPC status from /dev/kmem as of Sun Aug 10 03:28:44 2003 T      ID     KEY        MODE     OWNER     GROUP NATTCH  SEGSZ  CPID  LPID Shared Memory: m       0 0x411c05c4 --rw-rw-rw-   root      root      0    348   420   420 m       1 0x4e0c0002 --rw-rw-rw-   root      root      1  61760   420   420 m       2 0x41200888 --rw-rw-rw-   root      root      1   8192   420   432 m       3 0x301c3e60 --rw-rw-rw-   root      root      3 1048576   957   971 m       4 0x501c7c07 --r--r--r--   root     other      1 1075095  1023  1023 m       5 0x00000000 --rw-------   root      root      2  65536  1308  1308 root@hpeos001[clockwatch] # 

This showed me that all was back to normal. If you are going to conduct such a test, please ensure that your system returns to a normal state before proceeding.