5.1 Managing and monitoring your Web infrastructure

< Day Day Up >

e-business transaction performance monitoring is important; however, it is equally important to ensure that the TMTP system itself, as well as the entire Web infrastructure, is running correctly. One of the prerequisite components for implementing TMTP is WebSphere Application Server, which in turn may rely on a prerequisite Web server, for example, IBM HTTP Server. Without these components up and running, the TMTP will not be accessible, or worse, will not work correctly. The same reason is true for the database support needed by TMTP.

The IBM Tivoli Monitoring products provide the basis for proactive monitoring, analysis, and automated problem resolution. A suite of solutions known as the "IBM Tivoli Monitoring for ..." products allow an IT department to provide management of the entire business system in a consistent way, from a central site, using an integrated set of tools.

This chapter contains multiple references to additional product documentation and other sources, such as Redbooks, which you are encouraged to refer to for further details. Please see "Related publications" on page 479 for a complete list of the referenced documents.

Note

At the time of the writing of this redbook, the publicly available version of IBM Tivoli Monitoring for Web Infrastructure does not support WebSphere Version 5.0.1. This support was being tested within IBM and was due to be released shortly after our planned publishing date.

5.1.1 Keeping Web and application servers online

The IBM Tivoli Monitoring for Web Infrastructure provides an enterprise management solution for both the Web and application server environments. The Proactive Analysis Components (PAC) that make up this product provide solutions that are integrated with other Tivoli management products. A comprehensive and fully integrated management solution can be rapidly deployed and provide a very attractive return on investment.

The IBM Tivoli Monitoring for Web Infrastructure currently focuses primarily on the performance and availability aspect of managing a Web infrastructure. The four proactive analysis components of the IBM Tivoli Monitoring for Web Infrastructure product provides similar management functions for the supported Web and application servers:

Monitoring for IBM HTTP Server
Monitoring for Microsoft Internet Information Server
Monitoring for Sun iPlanet Server
Monitoring for WebSphere Application Server

The following sections provide information on how to set up and customize IBM Tivoli Monitoring for Web Infrastructure to ensure performance and availability of the Tivoli Web Site Analyzer application.

We will focus on the monitoring for the WebSphere Application Server. For the other Web severs, refer to the redbook Introducing IBM Tivoli Monitoring for Web Infrastructure, SG24-6618.

5.1.2 ITM for Web Infrastructure installation

In order to install IBM Tivoli Monitoring for Web Infrastructure, you need to complete the following steps:

Plan your management domain.
Check the prerequisite software and patches.
Choose the installation options.
Verify the installation.

For all these steps, refer to the IBM Tivoli Monitoring for Web Infrastructure Installation and Setup Guide V5.1.1, GC23-4717 or the redbook Introducing IBM Tivoli Monitoring for Web Infrastructure, SG24-6618. These publications contain all the information you need to set up IBM Tivoli Monitoring for Web Infrastructure, including the prerequisites needed to install the product.

As a prerequisite to ensure the availability of TMTP, we have to ensure the availability of the WebSphere Application Server and the IBM HTTP Server.

IBM WebSphere Application Server

These are the prerequisites you need on the WebSphere Application Server system:

IBM WebSphere Application Server Version 4.0.2 or higher.
An operational Tivoli Endpoint.
WebSphere Administration Server must be installed on the same system as the Tivoli endpoint.
Java Runtime Environment Version 1.3.0 or higher.
Monitoring at the IBM WebSphere Application Server must be enabled.

Java Runtime Environment

IBM Tivoli Monitoring for Web Infrastructure requires that the endpoints have Java Runtime Environment (JRE) Version 1.3.0 or higher installed. If a Java Runtime Environment currently is not installed on the endpoint, one can be installed from the IBM Tivoli Monitoring product CD. You can install JRE either manually or by running the wdmdistrib -J command, or by using the Tivoli Software Installation Service (SIS).

If you have just installed Java Runtime Environment or if you have an existing Java Runtime Environment, you need to link it to the IBM Tivoli Monitoring using the DMLinkJre task from the IBM Tivoli Monitoring Tasks TaskLibrary.

Note

For IBM WebSphere Application Server, you must use the IBM WebSphere Application Server's JRE.

Monitoring at the IBM WebSphere Application Server

The following details apply to any systems hosting IBM WebSphere Application Server that you want to manage with IBM Tivoli Monitoring for WebSphere Application Server:

IBM Tivoli Monitoring for WebSphere Application Server supports only one installation of WebSphere Application Server on each host system.
If security is enabled for IBM WebSphere Application Server, you should create a security properties file for the wscp client so that it can be authenticated by the server. You can copy the existing sas.client.props file in the $WAS_HOME/Properties directory ($WAS_HOME is the directory where you have installed your WebSphere Application Server) to sas.wscp.props and edit the following lines:
```
 com.ibm.CORBA.loginSource=properties com.ibm.CORBA.loginUserid=<userid> com.ibm.CORBA.loginPassword=<password> 
```
where <userid> is the IBM WebSphere Application Server user ID and <password> is the password for the user.
If you are using a non-default port for IBM WebSphere Application Server, you need to change the configuration of the endpoint in order to communicate with the IBM WebSphere Application Server object. You can do this by changing the port setting in the sas.wscp.props file. You can create the file in the same way as mentioned above and then add the following line:
```
 wscp.hostPort=<port_number> 
```
where <port_number> is the same value specified for property com.ibm.ejs.sm.adminServer.bootstrapPort in $WAS_HOME/bin/admin.config, where $WAS_HOME is the directory where you have installed your WebSphere Application Server.

To monitor performance data for your IBM WebSphere administration and application servers, you must enable IBM WebSphere Application Server to collect performance data. Each performance category has an instrumentation level, which determines which counters are collected for the category. You can change the instrumentation levels using the IBM WebSphere Application Server Resource Analyzer. On the Resource Analyzer window, you need to do the following:

Right-click on the application server instance, for example, WebSiteAnalyzer, and choose Properties, click on the Services tab and select Performance Monitoring Settings from the pop-up menu to display the Performance Monitoring Settings window.
Select Enable performance counter monitoring.
Select a resource and choose None, Low, Medium, High or Maximum from the pop-up icon. The color associated with the chosen instrumentation level is added to the instrumentation icon and all subordinate instrumentation levels.
Click OK to apply the chosen setting or Cancel to undo any changes and revert to the previous setting.

Table 5-1 lists the minimum monitoring levels for the IBM Tivoli Monitoring for Web Infrastructure WebSphere Application Server Resource Models.

Table 5-1: Minimum monitoring levels WebSphere Application Server
Resource Model	Monitoring setting	Minimum monitoring level
EJBs	Enterprise Beans	High
DB Pools	Database Connection Pools	High
HTTP Sessions	Servlet Session Manager	High
JVM Runtime	JVM Runtime	Low
Thread Pools	Thread Pools	High
Transactions	Transaction Manager	Medium
Web Applications	Web Applications	High

You should enable the Java Virtual Machine Profile Interface (JVMPI) to improve performance analysis. The JVMPI is available on the Windows, AIX, and Solaris platforms. However, you do not need to enable JVMPI data reporting to use the Resource Models included with IBM Tivoli Monitoring for WebSphere Application Server.

IBM HTTP Server

For the prerequisites needed to monitor the IBM HTTP Server, refer to IBM Tivoli Monitoring for Web Infrastructure Apache HTTP Server User's Guide Version 5.1, SH19-4572.

5.1.3 Creating managed application objects

Before you start to manage Web server resources, they must first be registered in the Tivoli environment. This registration is achieved by creating specific Web Server objects in any policy region. When installing IBM Tivoli Monitoring for Web Infrastructure, a default policy region corresponding to the IBM Tivoli Monitoring for Web Infrastructure module is automatically created. For the WebSphere Application Server module, this policy region is named Monitoring for WebSphere Application Server.

Note

Normally, managed application objects are created in the default policy regions. If you want to create the managed application objects in a different policy region, you must first add the relevant IBM Tivoli Monitoring for Web Infrastructure managed resource to the list of resources supported by the specific policy region.

The WebSphere managed application objects are created differently from the other Web server objects. In order to manage WebSphere Application Servers, two types of WebSphere managed application objects need to be defined:

WebSphere Administration Server managed application object
WebSphere Application Server managed application object

The WebSphere Administration Server managed application object must be created before the WebSphere Application Server managed application object.

You can create the managed application object for the WebSphere Server in three different ways:

Using the Tivoli desktop, in which case you need to follow these two steps:
1. Create the WebSphere Administration Server managed application object by selecting Create → WSAdministrationServer in the policy region, which will open the dialog shown in Figure 5-1 on page 159.
  
  Figure 5-1: Create WSAdministrationServer
2. Create the WebSphere Application Server managed application object by selecting Create → WSApplicationServer in the policy region. The dialog in which you can specify the parameters for the managed application object is shown in Figure 5-2 on page 160.
  
  Figure 5-2: Create WSApplicationServer
By using the discovery task Discover_WebSphere_Resource in the TaskLibrary WebSphere Application Server Utility Tasks, both objects will be created automatically for you. When starting the task, supply the parameters for discovery in the dialog, as shown in Figure 5-3 on page 161.

Figure 5-3: Discover WebSphere Resources
Run the appropriate command from the command line:
```
 wWebshpere -c 
```
Note
This method can only be used to create the WebSphere Application Server managed application object.

For all the specified parameters, commands, and the appropriate descriptions, refer to the IBM Tivoli Monitoring for Web Infrastructure Reference Guide Version 5.1.1, GC23-4720 and the IBM Tivoli Monitoring for Web Infrastructure: WebSphere Application Server User's Guide Version 5.1.1, SC23-4705.

If all the parameters supplied to the Tivoli Desktop, the command line, or the task are correct, the managed server objects icons shown in Figure 5-4 on page 162 are added to the policy region.

click to expand
Figure 5-4: WebSphere managed application object icons

5.1.4 WebSphere monitoring

The following section will outline tasks needed to activate monitoring of the availability and performance of the Tivoli Web Site Analyzer application's operational environment with IBM Tivoli Monitoring for Web Infrastructure.

Resource Models

A Resource Model is used to monitor, capture, and return information about multiple resources and applications. When adding Resource Models to a profile, these are chosen based on the type of resources that are being monitored.

WebSphereAS is the abbreviated name of the IBM Tivoli Monitoring category of the IBM WebSphere Application Server Resource Models. It is used as an identifying prefix.

Planning

The following list gives the indicators available in the Resource Models provided with the Tivoli PAC for WebSphere Application Server:

WebSphereAS Administration Server Status: Administration server is down, occurs when the status of the WebSphere Application Server administration server is down.
WebSphereAS Application Server Status: Application server is down, occurs when the status of the WebSphere Application Server application server is down.
WebSphereAS DB Pools:
- Connection pool timeouts are too high, which occur when the database connection timeout exceeds a predefined threshold.
- DB Pools avgWaitTime is too high, which occurs when the average time required to obtain a connection in the database connection pool exceeds the predefined threshold.
- Percent connection pool used is too high, which occurs when the percentage of database connection in use is higher than a predefined threshold (assuming you have sufficient network capacity and database availability, you might need to increase the size of the database connection pool).
WebSphereAS JEJB:
- Enhanced Java Bean (EJB) performance, either gathered at the EJB or application server (EJB container) level, which occurs when the average method response time (ms) exceeds the response time threshold. The load is also reported by concurrent active EJB requests, and throughput is measured by the EJB request rate per minutes.
- EJB exceptions, either gathered at the EJB or application server (EJB container) level, which occur when a specified percentage of EJBs are being discarded instead of returned to the pool. The returns discarded (as a percentage of those returned to the pool) exceeded the defined threshold. If you receive this indication, you may need to increase the size of your EJB pool.
WebSphereAS HTTP Sessions: LiveSessions is too high, which occurs when the number of live sessions exceeds the predefined "normal" amount for an application.
WebSphereAS JVM Runtime: Used JVM memory is too high, which occurs when the percentage of used JVM memory exceeds a defined percentage of the total available memory.
WebSphereAS Thread Pools: Thread pool load, which occurs when the ratio of active threads to the size of the thread pool exceeds the predefined threshold.
WebSphereAS Transaction:
- The recent transaction response time is too high, which occurs when the average transaction response time exceeds a predefined threshold.
- The timed-out transactions are too high, which occur when transactions exceed the time-out limit and are being terminated (a maximum ratio for timed-out transactions to total transactions).
WebSphereAS Web Applications:
- Servlet/JSP errors, either at the application server or Web application or servlet level, which occurs when the number of servlet error passes a predefined normal amount of errors for the application.
- Servlet/JSP performance, either at the application server or Web application or servlet level, which occurs when the servlet response time exceeds the predefined monitoring threshold.

During the initial deployment on any Resource Model of IBM Tivoli Monitoring for Web Infrastructure, we recommend using the default values shown in Table 5-2. The following definitions will help you understand the table.

Table 5-2: Resource Model indicator defaults
	Indication	Cycle time	Threshold	Occurrences/Holes
*WebSphereAS Administration Server Status*
	Administration Server is down.	60s	down	1/0
*WebSphereAS Application Server Status*
	Application Server is down.	60s	down	1/0
*WebSphereAS DB Pools*
	Connection pool timeouts are too high.	90s	0	9/1
	DB Pool avgWaitTime is too high.	90s	250ms	9/1
	Percent connection Pool used is too high.	90s	90	9/1
*WebSphereAS EJB*
	EJB performance (data gathered at EJB level).	90	0	9/1
	EJB performance (data gathered at application server, EJB container, and level).	90	0	9/1
	EJB exceptions (data gathered at EJB level).	90s	50%	9/1
	EJB exceptions (data gathered at application server, EJB container, and level).	90s	50%	9/1
*WebSphereAS HTTP Sessions*
	LiveSessions is too high.	180s	1000	9/1
*WebSphereAS JVM Runtime*
	Used JVM memory is too high.	60s	95%	1/0
*WebSphereAS Thread Pools*
	Thread Pool load.	180s	95%	9/1
*WebSphereAS Transactions*
	Recent transaction response time is too high.	180s	1000ms	9/1
	Timed-out transactions are too high.	180s	2%	9/1
*WebSphereAS Web Applications*
	Servlet/JSP errors (at application server level).	90s	0	9/1
	Servlet/JSP errors (at Web application level.	90s	0	9/1
	Servlet/JSP errors (at servlet level).	90s	0	9/1
	Servlet/JSP performance (at application server level).	90s	750ms	9/1
	Servlet/JSP performance (at Web application level.	90s	750ms	9/1
	Servlet/JSP performance (at servlet level).	90s	750ms	9/1

Number of Occurrences	Specifies the number of consecutive times the problem occurs before the software generates an indication.
Number of Holes	Determines how many cycles that do not produce an indication can occur between cycles that do produce an indication.

Deployment

After deciding which Resource Models and indications you need, you have to deploy the monitors. This means you have to:

Create profile managers and profiles. This will help organize and distribute the Resource Models.

A monitoring profile may be regarded as a group of customized Resource Models that can be distributed to a managed resource in a profile manager. The profile manager has to be created first with the wcrtprfmgr command or from the Tivoli desktop. After this, you can create the profile, which should be a Tmw2kProfile (must be included in the managed resources of the policy region), with the wcrtprf command or from the Tivoli desktop.
Add subscribers to the profile managers.

The subscribers of a profile manager determine which systems will be monitored when the profile is distributed. You can do this with either the wsub command or from the Tivoli desktop. The subscribers for IBM Tivoli Monitoring for Web Infrastructure would be the managed application objects that were created in 5.1.3, "Creating managed application objects" on page 158.
Add Resource Models.

We recommend that you group all of the Resource Models to be distributed to the same endpoint or managed application object in a single profile. You can now add the Resource Models with the parameters you have chosen to the profiles. You can do this by using either the wdmeditprf command or the Tivoli desktop, as shown in Figure 5-5 on page 167.

Figure 5-5: Example for an IBM Tivoli Monitoring Profile
Distribute the profiles.

You can do this by either using the wdmdistrib command or the Tivoli desktop.

Tivoli Enterprise Console adapter

By default, all the Resource Models will send an event to the Tivoli Event Console event management environment whenever a threshold is violated. These events may be used to trigger actions based on rules stored in the TEC Server.

Another possible way to send events to the TEC environment is directly from the WebSphere Application Server using the IBM WebSphere Application Server Tivoli Enterprise Console adapter. This adapter is used to forward native WebSphere Application Server messages (SeriousEvents) to the Tivoli Enterprise Console. These messages may have the following severity codes:

FATAL
ERROR
AUDIT
WARNING
TERMINATE

The Tivoli Enterprise Console adapter is also self-reporting; you can see adapter status events in the WebSphere Application Server console.

A task is created during the installation of the product in the WebSphere Event Tasks TaskLibrary. This task, Configure_WebSphere_TEC_Adapter, is used to configure the adapter. Before executing this task, make sure that the IBM WebSphere Administration Server is running. Then you have to configure which messages you want to be forwarded to the Tivoli Enterprise Console.

The WebSphere Event Tasks TaskLibrary also includes two tasks with which you can start and stop the Tivoli Enterprise Console adapter. The task names are:

Start_WebSphere_TEC_Adapter
Stop_WebSphere_TEC_Adapter

5.1.5 Event handling

Tivoli Enterprise Console (TEC) has been designed to receive events from multiple sources and process them in order to correlate and aggregate them, and issue predefined (corrective) actions based on the processing. TEC works on the basis of events and rules.

TEC events are defined in object-oriented definition files called BAROC files. These events are defined hierarchically according to their type. Each event type is called an event class. When TEC receives an event, it parses the event to determine the event class and then apply the class definition to parse the rest of the event; when the parsing is successful, the event is stored in the TEC database.

When a new event is stored, a timer expires, or a field (known in TEC terminology as a slot) has changed, TEC evaluates a set of rules to be applied to the event. These rules are stored in ruleset files, which are written in the Prolog language. When a matching rule is found, the action part of the rule is executed. These rules enable events to be correlated and aggregated. Rules also enable automatic responses to certain conditions; usually, these are corrective actions.

In the IBM Tivoli Monitoring for Web Infrastructure perspective, Web- and application server specific events are generated by the Resource Models provided by each of the IBM Tivoli Monitoring for Web Infrastructure modules. These events are defined in TEC and a set of predefined rules exists to correlate and process the events.

To set up a TEC environment capable of receiving Web and application server related events from IBM Tivoli Monitoring for Web Infrastructure environment, at least the following components have to be installed:

Tivoli Enterprise Console Server Version 3.7.1
Tivoli Enterprise Console Version 3.7.1
Tivoli Enterprise Console User Interface Server Version 3.7.1
Tivoli Enterprise Console Adapter Configuration Facility Version 3.7.1

TEC also uses a RDBMS system in which events are stored. Please refer to the IBM Tivoli Enterprise Console User's Guide Version 3.8, GC32-0667 for further details on TEC installation and use.

IBM Tivoli Monitoring for Web Infrastructure events and rules

In order to define the IBM Tivoli Monitoring for Web Infrastructure related events and rules to the TEC, the proper definition files have to be imported into the TEC environment. The IBM Tivoli Monitoring for Web Infrastructure events and rules are described in files that have .baroc and .rls file extensions. All the files can be found in the directory in which the Tivoli Enterprise Console server code is installed (in the subdirectory bin/generic_unix/TME®).

The definition files for the IBM Tivoli Monitoring for WebSphere Application Server events are documented in the subdirectory WSAPPSVR in the following BAROC files:

itmwas_dm_events.baroc	Definitions for the events originated from all the Resource Models
itmwas_events.baroc	Definitions of events forwarded to the TEC directly from the WebSphere Application Server and the Tivoli Enterprise Console adapter

For the IBM Tivoli Monitoring for WebSphere Application Server events, three different rulesets are supplied in the subdirectory WSAPPSVR:

itmwas_events.rls	Handles events that originate directly from the WebSphere Application Server Tivoli Enterprise Console adapter
itmwas_monitors.rls	Handles events that originate from Resource Models
itmwas_forward_tbsm.rls	Handles events that are forwarded to Tivoli Business System Manager

Tivoli provides for all the IBM Tivoli Monitoring for Web Infrastructure solutions definition files and ruleset files. They are located in the appropriate subdirectories. For documentation regarding these files, please refer to the appropriate User's Guides for the IBM Tivoli Monitoring for Web Infrastructure modules.

For further information on how to implement the classes and rule files, refer to the IBM Tivoli Enterprise Console Rule Builder's Guide Version 3.8, GC32-0669.

5.1.6 Surveillance: Web Health Console

You can use the IBM Tivoli Monitoring Web Health Console to display, check, and analyze the status and health of any endpoint, where monitoring has been activated by distributing profiles with Resource Models. The endpoint status reflects the state of the endpoint displayed on the Web Health Console, such as running or stopped. Health is a numeric value determined by Resource Model settings. The typical settings include required occurrences, cycle times, thresholds, and parameters for indications. These are defined when the resource model is created. You can also use the Web Health Console to work with real-time or historical data from an endpoint that is logged to the IBM Tivoli Monitoring database.

You can connect the Web Health Console to any Tivoli management region server or managed node and configure it to monitor any or all of the endpoints that are found in that region. The Web Health Console does not have to be within the region itself, although it may.

To connect to the Web Health Console, you need access to the server on which the Web Health Console server is installed and the Tivoli Management Region on which you want to monitor the Health Console. All user management and security is handled through the Tivoli management environment. This includes creating users and password as well as assigning authority.

To activate the online monitoring of the health of a resource, you have to log in to the Web Health Console. This may be achieved by performing the following steps:

Open your browser and type the following text in the address field:

/dmwhc">http://<server_name>/dmwhc

where <server_name> is the fully qualified host name or IP address of the server hosting the Web Health Console.
Supply the following information:

User

Tivoli user ID

Password

Password associated with the Tivoli user ID

Host name

The managed node to which you want to connect
The first time you log in to the Web Health Console, the Preferences view is displayed. You must populate the Selected Endpoint list before you can access any other Web Health Console views. When you log in subsequently, the endpoint list is loaded automatically.
Select the endpoints that you want to monitor and choose the Endpoint Health view. This is the most detailed view of the health of an endpoint. In this view, the following information is displayed:
1. The health and status of all Resource Models installed on the endpoint.
2. The health of the indications that make up the Resource Model and historical data.

After setting up the Web Health Console, you are able to display the health of a specific endpoint; to view the data, use the theoretical view option. Figure 5-6 shows an example of real-time monitoring of an WebSphere Application Server.

click to expand
Figure 5-6: Web Health Console using WebSphere Application Server

For detailed information on setting up and working with the Web Health Console, refer to the IBM Tivoli Monitoring User's Guide V5.1.1, SH19-4569.

< Day Day Up >

User	Tivoli user ID
Password	Password associated with the Tivoli user ID
Host name	The managed node to which you want to connect