Managing Your WebLogic Servers Using the Node Manager

When you have a production WebLogic domain consisting of distributed managed servers, it is important for critical tasks such as starting, stopping, killing, and restarting the managed servers to be performed in the most efficient manner. Rather than creating your own scripts to perform these tasks , you can use the Node Manager.

The Node Manager, as shown in Figure 24.23, is a standalone Java program that can be leveraged to improve the availability of your distributed managed servers within a WebLogic domain by performing the following automated administration tasks:

Figure 24.23. The architecture and operation of a Node Manager within a WebLogic domain.

• Monitor the quality of health of one or managed servers and restart those managed servers that are deemed to be in a Failed state.

• Provide auto detection and restart of crashed managed servers (through system crashes, hardware reboots, or server failures).

• Remotely start up, shut down, or force shutdown (kill) individual managed servers, as well as a whole domain or cluster of managed servers.

Note

You cannot automatically restart managed servers that were booted without using the Node Manager (for example, servers started directly from scripts or the command line). To automatically restart managed servers, you must configure and start the servers using the Node Manager.

You can use the Administration Console to shut down a managed server, but you cannot use it to start a managed server without employing a Node Manager.

As illustrated in Figure 24.23,

• The Node Manager is a separate Java process from WebLogic Server and runs on the same physical machine as the WebLogic Server you want to remotely manage.

• Communication is dispatched from the administration server to the Node Manager using the Administration Console or weblogic.Admin utility.

• A Node Manager can monitor the health of one or more managed servers running on one machine, even if those managed servers belong to different WebLogic domains.

• The Node Manager can be run as a Windows service or Unix daemon to ensure it is always running, even after a machine is rebooted. This is imperative for a production environment.

• All communications between the administration server and the managed servers via the Node Manager occur using SSL; hence, all managed servers must have SSL enabled.

  Note

  The Node Manager is specifically meant for operations only on managed servers. Hence, you cannot use the Node Manager to start, shut down, or kill the administration server.

  
  

• The operations and events of the Node Manager are written to the Node Manager log file.

• The Node Manager software must be at the highest release of all the managed servers it is monitoring on that machine node. Also, you can use it only with WebLogic Server software that is version 6.1 SP2 and higher.

Configuring the Node Manager on a Machine Node

Now that you have learned the objectives and reasons for using a Node Manager within a distributed WebLogic domain, the next step is to learn how to configure and start the Node Manager.

The steps in the following sections guide you through the process of configuring a Node Manager on the machine nodes where your managed servers are located.

Step 1: Configuring Your Trusted Hosts

To ensure the Node Manager receives only legitimate and trusted communication, you need to explicitly specify the network address of the administration server machines from which the Node Manager node will be receiving connection requests .

You can do so by editing the nodemanager.hosts text file located in the WL_HOME\ common\nodemanager\config directory and entering the IP address or DNS names of the administration servers that will be initiating Node Manager connections. You need to perform this task only for the machines that will be hosting managed servers.

If you specify a DNS name for an administration server in nodemanager.hosts , you will need to enable reverse DNS lookup in the Node Manager startup script using the following command-line argument:

 -Dweblogic.nodemanager.reverseDnsEnabled=true 

Note

By default, reverse DNS lookup is disabled for a Node Manager.

Step 2: Configuring SSL Communication Between the Administration Server and Managed Servers

All communications between administration servers and the Node Manager use the SSL protocol, which requires the use of the same public key infrastructure as WebLogic Server 7: a certificate as well as a private key. To enable SSL, you have two options:

• You can use the default WebLogic Server demonstration key and certificate files ( demokey.pem and democert.pem ), which will help you to quickly enable SSL communication in a development or test environment.

• You can obtain your own digital certificates and private keys, which are required within a production WebLogic Server deployment environment.

Because using the default WebLogic Server demonstration key and certificate files is the fastest and easiest roadmap for configuring SSL within a WebLogic domain, use the same steps to enable SSL as discussed in the section "Enabling the Administration Port" earlier in this chapter. The administration server SSL configuration applies to the domain as a whole.

Step 3: Configuring the Node Manager Machines

Because the administration server is the client of the Node Manager, it must be configured to recognize the following:

• The managed servers that will be managed and monitored using the Node Manager

• The machine nodes that host the managed servers in the domain

• The listen address and port number it will use to connect to the Node Manager process running on machine node

The first two points can be accomplished by following the steps to configure the machines and servers using the Administration Console, described earlier in this chapter. These steps ensure you have mapped the managed servers to their respective machine nodes.

To configure the listen address and port number the administration server will use to connect to a Node Manager process, follow these steps:

1. Select the machine on which the Node Manager will be hosted in the left pane of the Administration Console.

2. Select the Configuration, Node Manager tab, and enter the following information:

   • Listen Address Enter the host name or IP address the Node Manager will use for incoming requests from the administration server. (The default is Localhost.)

   • Listen Port Enter the port number on which the Node Manager will listen for requests. (The default is 5555.)

3. Click Apply.

Step 4: Specifying the Managed Server's Startup Information

You should have a consistent set of startup arguments for each managed server in your domain, regardless of the startup method you use. For a Node Manager to start a managed server successfully, it must receive that managed server's startup arguments and environment information from the administration server.

To enter these startup arguments and environment information for each managed server using the Administration Console, follow these steps:

1. In the left pane, select Servers; then select the name of the managed server that will be remotely started using the Node Manager.

2. Select the Configuration, Remote Start tab in the right pane to display the managed server's startup configuration.

3. As shown in Figure 24.24, edit the following attributes for starting the respective managed server:

   Figure 24.24. Configuring a managed server's startup information via the Node Manager.

   

   • Java Home Enter the fully qualified path of the Java home directory (Java Runtime Environment, or JRE), which will be used to start the managed server. For example, if you are using the JRE provided by BEA, you can enter < drive >:\bea\jdk131 .

   • BEA Home Enter the root directory of the BEA-installed products and licensefor example, < drive >:\bea\ .

   • Root Directory Enter the root directory of the managed server domainfor example, < drive >:\bea\user_projects\domain_name . This will also serve as the directory for lookups by the applications deployed in the respective managed server.

   • Class Path Specify the CLASSPATH variable to use when starting the managed servers, which must include the weblogic.jar and weblogic_sp.jar filesfor example,

      drive>:\bea\weblogic700\server\lib\weblogic.jar drive>:\bea\weblogic700\server\lib\weblogic_sp.jar 

   • Arguments Specify any additional arguments to pass to the JVMfor example, the JAVA_OPTIONS arguments specified in the managed server's startup script.

   • Security Policy File Enter the fully qualified path to the weblogic.policy file.

   • Username Enter the name of a user who has the credentials to start the managed server.

   • Password Leave the password as is, unless you need to change it.

   Caution

   Do not add a \ at the end of each of the paths because it will cause an error starting the respective managed server.

4. Click Apply.

You have now completed all the steps required to configure the Node Manager and the managed servers that will be managed via the Node Manager.

Starting the Node Manager Process

You can start the Node Manager Java program by using one of the following methods :

• Manually, using the java command at the command prompt.

• Invoking a startup script.

• As a Windows service or Unix daemon process, which enables the Node Manager to be automatically started whenever its hosting machine is booted. This is the recommended startup method for production WebLogic environments.

Because you have to specify the same Node Manager environment variables and properties regardless of the startup method you select, you should use the startup script initially. This approach allows you to easily proof your Node Manager startup settings. When you have the Node Manager fully operational with the ability to start, monitor, and shut down a managed server via the Administration Console, you can implement a Windows service or Unix daemon with a proven set of Node Manager environment variables and properties.

The following sections discuss how to configure and start the Node Manager using a startup script and then how to create a Node Manager Windows service.

Starting the Node Manager Using a Startup Script

The default Node Manager startup script, named startNodeManager.cmd , is located in the WL_HOME\ server\bin directory, where WL_HOME is the WebLogic Server installation directory ( C:\bea\weblogic700 by default). To start the Node Manager, you need to review and edit the environment variables and properties (startup arguments) set in this file using any text editor. To assist in this task, you can use the Node Manager and WebLogic environment variables and properties described in Table 24.3 as a guide for correctly configuring your Node Manager startup script.

Table 24.3. The Environment Variables and Properties for a Node Manager Startup Script

Node Manager Startup Variables and Properties	Description
%NODEMGR_HOME%	The Node Manager working directory for storing output and log files
%JAVA_HOME%	The root directory of the installed JDK used to start the Node Manager
%DOMAIN%	The root directory of the managed server domain
%MEM_ARGS%	The memory arguments you want to pass to the JVM
-Dbea.home	The directory where the BEA license files are located
-Djava.security.policy	The path to the WebLogic security policy file
-Dweblogic.nodemanager.sslHostNameVerificationEnabled	Determines if the Node Manager performs host name verification
-Dweblogic.nodemanager.javaHome	The Java home directory that Node Manager uses to start the managed servers
-Dweblogic.nodemanager.reverseDnsEnabled	Determines if DNS names will be used instead of an IP address for the listen port
Dweblogic.ListenAddress	The address on which Node Manager listens for connection requests
Dweblogic.nodemanager.keyFile	The path to the private key file to use for SSL communication
Dweblogic.security.SSL.trustedCAKeyStore	The path to the KeyStore that contains the certificates of trusted authorities
Dweblogic.nodemanager.certificateFile	The path to the certificate file used for SSL authentication

The following are sample values that can be assigned to the environment variables and properties described in Table 24.3:

  WL_HOME  =C:\bea\weblogic700 NODEMGR_HOME=%  WL_HOME  %\common\nodemanager JAVA_HOME=C:\bea\jdk131_03 DOMAIN=c:\bea\user_projects MEM_ARGS=-Xms32m -Xmx50m -Djava.security.policy=%  WL_HOME  %\ server\lib\weblogic.policy -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.nodemanager.javaHome=%JAVA_HOME% -Dweblogic.nodemanager.sslHostNameVerificationEnabled=FALSE weblogic.nodemanager.reverseDnsEnabled=TRUE -Dweblogic.ListenAddress=SOCRATES -Dweblogic.ListenPort=7266 -Dweblogic.nodemanager.javaHome=%JAVA_HOME% -Dweblogic.nodemanager.keyFile=c:\bea\user_projects\objectmind\demokey.pem -Dweblogic.nodemanager.certificateFile=c:\bea\user_projects\objectmind\democert.pem 

After you appropriately edit the Node Manager startup arguments, you can start the Node Manager from the command line by executing the startNodeManager.cmd command, as shown in Figure 24.25.

Figure 24.25. The startup console for the Node Manager.

Creating a Node Manager Windows Service

The script that is used to create a Node Manager service, named installNodeMgrSvc.cmd , is located in the WL_HOME \server\bin directory. You need to edit this file before it can be invoked, similar to the startNodeManager.cmd script, to ensure the Node Manager's service name and startup arguments are set according to your WebLogic environment.

You also need to reflect the changes you make in the uninstallNodeMgrSvc.cmd file, which is the script used to uninstall the Node Manager service.

After the Node Manager service is created, you can start and stop it using the Windows Service Manager.

Remotely Starting the Managed Servers

After a Node Manager is successfully started on a machine, you can engage in starting one, multiple, or a cluster of managed servers on that machine in the context of a WebLogic domain using the Administration Console.

Note

All managed servers you intend to remotely start must be configured as described earlier in the section "Step 4: Specifying the Managed Servers Startup Information."

To start a managed serverfor example, mServer that was created earlier in this chapterusing the Administration Console, follow these steps:

1. From the Servers node, select the name of the configured server you want to start (mServer).

2. Select the Control, Start/Stop tab in the right pane.

3. Click the Start This Server link.

4. Click Yes or No to verify your selection.

When you start a managed server using the Administration Console via the Node Manager, you can view the startup messages using the Details tab, as shown in Figure 24.26.

Figure 24.26. Reviewing the startup messages for a remotely started managed server.

To stop the server, you can perform the same steps except that you select the shutdown option. You can also use the Administration Console to start and stop all the managed servers in a cluster or in a domain, as long as the managed servers are configured for remote start.

The Node Manager always starts or restarts a managed server in its last runtime state, regardless of the startup mode you have specified.

Reviewing the Node ManagerAssociated Log Files

If you incur any problems starting a Node Manager or while starting or stopping managed servers via the Node Manager, reviewing the log files associated with Node Manager should be your first task in diagnosing the root cause of the problem. These files hold valuable information that will help you isolate the problem.

The log files related with a Node Manager and the managed servers it starts are located on the machine that hosts the Node Manager within the NodeManagerLogs directory, which is within the directory specified by the %NODEMGR_HOME% variable in the startNodeManager.cmd script.

The Node Manager Startup Log Files

Each time a Node Manager is started, the related startup and status messages are written to a unique log file named NodeManagerInternal_ timestamp , which is located in the NodeManagerInternal subdirectory of the NodeManagerLogs directory. You should view this file using a text editor to ensure there are no problems with the startup or operation of the Node Manager.

Tip

Because the NodeManagerInternal_ timestamp file is created each time you start a Node Manager, whether or not the startup process is successful, it is good practice to delete the log files that are no longer required.

The Managed Server Log Files

For each managed server that is successfully started using the Node Manager, a separate log file subdirectory is created under the NodeManagerLogs directory. The naming convention used for these managed server directories is domain _ server , where domain represents the name of the domain and server represents the name of the managed server.

The files generated for each started managed server in this directory are as follows :

• Managed_Server_Name_pid Stores the process ID of the managed server to enable the administration server to kill the managed server via the Node Manager.

• config.xml Is the configuration property file that is used to start the managed server. This subset of the domainwide config.xml file is copied from the administration server upon the managed server's connection.

• Managed_Server_Name_ log Stores the startup information of the managed server.

• Managed_Server_Name_ error_log Stores any error messages related to the managed server.

Caution

BEA recommends not deleting this file when a managed server is running.

Alternatively, you can also view the log files on the administration server. The NodeManagerClientLogs directory, which is located in the root directory of the administration server, contains a log file subdirectory for each managed server that is started via the Node Manager.

Tip

If you are having problems starting a managed server via the Node Manager for the first time, there is a possibility that no managed server log files will be generated because this depends on how far the startup process goes before it fails. For this reason, you should initially review the Node Manager log ( NodeManagerInternal_ timestamp ) if you have problems starting a managed server.

Monitoring the Health of Your WebLogic Domain Using the Node Manager

The Self-Health monitoring capability is a new feature of the WebLogic Server; it enables decoupled critical subsystems such as the WebLogic Server Kernel, JMS, and JTA to provide feedback on their health to determine the overall health of a WebLogic Server instance.

When a WebLogic Server starts, all self-healthenabled subsystems register themselves with their Health Monitoring System. Rather than the WebLogic Server determining the health of these subsystems, which can be a resource/performance overhead, each of the registered subsystems is responsible for internally and asynchronously monitoring its own health based on criteria specific to its subsystem. If at any point a self-healthenabled subsystem determines it can no longer operate in a consistent and reliable manner, it registers a Failed health state with its associated runtime MBeanfor example, the JMSRuntimeMBean for a JMS subsystem.

To determine the overall health of a WebLogic Server instance, the Health Monitoring System periodically (the default is 180 seconds) queries the health state attribute of the runtime MBeans associated with self-healthenabled subsystems. If any of these subsystems are found to be in a Failed state, the overall state of the associated WebLogic Server instance is also considered to be in a Failed state because the overall health of a WebLogic Server instance is determined as the sum of all the registered subsystems.

To improve the reliability and availability of your WebLogic domain, you can use the Health Monitoring System in conjunction with the Node Manager to ensure managed servers that are deemed to be in a Failed state are immediately shut down and, if required, restarted again. A Node Manager can deem a managed server to be in Failed state for a number a reasons; the following are some examples:

• The managed server has crashed.

• The managed server is hung and hence not responding.

• The managed server is not responding to any health check queries, which could be because it is busy.

• One or more critical self-health monitoring services have declared a Failed state.

The only caveat is that the Node Manager can monitor the health state of only those managed servers it has started.

To configure a Node Manager to monitor a managed server's health, you need to edit the managed server's Health Monitoring attributes, which you can do using the Administration Console as follows:

1. From the Administration Console, select the name of the managed server you want to configure for automated health monitoring using the Node Managerfor example, mServer.

2. Select the Configuration, Health Monitoring tab.

3. From the displayed managed server's automated restart and monitoring attributes, as shown in Figure 24.27, enable and edit those that pertain to your health monitoring requirements using the attribute descriptions in Table 24.4 as a guide.

   Figure 24.27. A managed server's self-health monitoring attributes.

   

4. Click Apply.

Table 24.4. The Node Manager Self-Health Monitoring Attribute Descriptions

Health Monitoring Attribute	Description
Auto Restart	This attribute should be enabled to ensure the Node Manager restarts a managed server that has been shut down. (This attribute is enabled by default.)
Auto Kill If Failed	This attribute enables the Node Manager to automatically kill a managed server after its health state has been deemed to be in a Failed state.
Restart Interval	If Auto Restart is enabled, this attribute specifies the length of time (seconds) during which the Node Manager will attempt to restart the managed server. The default is 3600 seconds or one hour . (This value should be a reflection of the duration of time before human intervention is a necessity.)
Max Restarts Within Interval	If Auto Restart is enabled, this attribute specifies the maximum number of times a Node Manager can restart the managed server during the Restart Interval. The default is 2. (If a managed server needs to be restarted more than twice during the Restart Interval, there is a high degree of probability the managed server will continue to fail, and the root cause of the problem will need to be immediately discovered .)
Health Check Interval	This attribute specifies the interval of time (seconds) between which periodic scans are performed by the Node Manager to verify whether the managed servers are in a Failed state. The default is 180 seconds. (The value should be set in accordance to the stability and criticality of the deployed application or applications on the managed server.)
Health Check Timeout	This attribute specifies the length of time (in seconds) the Node Manager will wait for a response to the health check query after which it will deem the monitored managed server to be in a Failed state. The default is 60 seconds. (This value should be based on the managed server's response time during its peak transaction period, which can prevent the server from being prematurely deemed to be in a Failed state.)
Restart Delay Seconds	This attribute specifies the duration of time the Node Manager will wait after a managed server has been restarted before it initiates its health monitoring activities. The default is 120 seconds. (You should set this attribute based on a proven time period for the successful start and the deployed application activity of a managed server.)

Note

Disabling the Auto Restart and enabling the Auto Kill If Failed attributes will cause a Node Manager to kill a Failed managed server if a Failed state is detected . However, the managed server will not be restarted.

Because a Node Manager can apply the Health monitoring and subsequent shutdown and restart activities only on those servers it starts, you need to shut down and then restart those servers that will be monitored via the Node Manager.