Network Configuration

Now that you know how the various J2EE services operate in a clustered environment and understand different cluster architectures, let's turn our attention to the network infrastructure needed to support WebLogic clustering. By understanding how WebLogic clusters interact with this infrastructure, you are in a better position to configure and optimize the cluster configuration. WebLogic uses two forms of network communication within a cluster:

• WebLogic instances within a cluster use multicast communication to broadcast heartbeats. These heartbeats verify the server's availability to other members of the cluster. The clustered JNDI service also relies on multicast communication to broadcast the availability of cluster-aware objects and services. Thus, multicast communication is vital for both high availability and load balancing.
• Socket communication is used in peer-to-peer communication between members of a cluster. This can occur when server-side objects access objects on a peer server, or when session-state information is replicated to a secondary server.

Each server listens for heartbeats broadcast by other servers in the cluster. This enables the server to detect any failure among its peers. In fact, each server broadcasts a unique heartbeat message every 10 seconds. If a server in the cluster misses three heartbeats of a peer, it marks that peer server as failed. Likewise, if a socket is closed unexpectedly by a peer server, the server that initiated the communication with the peer marks it as failed. So, both multicast heartbeats and peer-to-peer sockets can be used to detect the operational status of members within a cluster. In both cases, the server marks the peer server as failed and then updates its local JNDI tree to remove all services published by that peer. This ensures that peer services will no longer take part in any failover and load-balancing considerations.

This behavior stresses the importance of a stable and properly configured network communication infrastructure. In order to set up a WebLogic cluster successfully, you need to configure the cluster and its members so that both forms of communication can take place reliably.

14.8.1 Multicast Communication

WebLogic uses IP multicasts for one-to-many communication among server instances of a cluster. IP multicast communication manifests in various ways:

• Broadcast of cluster heartbeats relies on multicasts. Each server instance in a cluster uses a multicast to broadcast a heartbeat message indicating its availability. Each server in the cluster monitors the heartbeats of fellow servers, and marks a fellow server as failed when that server cannot detect any heartbeats.
• WebLogic's clustered JNDI service also uses multicast messaging. For example, when a clusterable EJB component is deployed to a server, the server advertises its availability to other members of the cluster by issuing a multicast broadcast to update their local JNDI trees.
• The implementation of WebLogic's multicast delivery to a JMS topic also depends on IP multicast. Here, each topic subscriber listens for a multicast broadcast of the JMS message.

Multicast addresses are IP addresses in the range 224.0.0.0 to 239.255.255.255. Messages sent as a multicast broadcast are not guaranteed to be delivered, or to be delivered in order. For this reason, the cluster infrastructure must allow for the possibility of missed messages. Multicast communication is sometimes difficult to set up, especially in older networks. If it isn't configured properly, and heartbeat messages are not being received in a timely fashion, clustered servers may erroneously mark each other as unavailable. You should adhere to the following guidelines when designing the network configuration for your clustered architecture:

Be careful when using network equipment within a WebLogic cluster

For instance, we recommend that you don't split a cluster across a firewall. This includes any other network equipment that may interrupt IP traffic and disrupt cluster communication by delaying or removing critical multicast messages, such as cluster heartbeats.

Ensure that other applications do not use the same multicast address and port

Other applications should not use the multicast address and port assigned to a WebLogic cluster. If other services within the cluster need to use multicast communication, assign a different multicast address and port for these services. Introducing external applications can cause the multicast buffer to overrun, resulting in lost messages. Multiple clusters may share the same multicast address and port without any problems.

Take care when configuring multicast across subnets

If a WebLogic cluster is distributed across multiple subnets, you need to ensure that the network is configured to do this properly and efficiently. This means configuring all routers and other equipment to propagate multicast packets properly. In these circumstances, the network latency also should be kept as low as possible, ensuring that multicast packets reach their destinations in no more than 200 to 300 milliseconds. In addition, the time-to-live (TTL) value for multicast packets should be set at the appropriate level for the subnets, ensuring that routers do not discard the packets prematurely. The TTL value for multicast packets indicates the number of subnets that a multicast packet can traverse before it will be discarded.

Avoid long-running garbage collection cycles

Ensure that your servers aren't busy with long-running garbage collection cycles. If your garbage collector runs for more than 10 seconds, for instance, a WebLogic instance may not get the opportunity to send a heartbeat. A change in your heap allocation and/or garbage collection strategy can help avoid this problem.

14.8.1.1 Multicast setup

In order to establish a cluster, you need to decide on a multicast cluster address and port. Using the Administration Console, you'll find that the Configuration/Multicast tab for a cluster holds all of the configuration settings for multicast setup. The address, port, and TTL values are absolutely essential settings for proper cluster configuration:

Multicast address

This address is used for all multicast communication within the cluster.

Multicast port

This port number is used for all multicast communication within the cluster.

Multicast TTL

The TTL value indicates the number of subnets that a multicast packet may traverse before it will be discarded.

Specifying a multicast address and port may not be enough. On a multihomed machine, it won't be obvious which network card WebLogic should use to handle all the multicast traffic. To resolve this problem, you need to set the Interface Address for each server. This is available in each server's Configuration/Cluster tab. Use this setting to specify the IP address of the NIC that will be used for all multicast traffic on that server.

Two additional low-level configuration options can help optimize multicast traffic:

Multicast Send Delay

This setting defines a delay (in milliseconds) before sending message fragments. This delay helps avoid buffer overflow at the OS level.

Multicast Buffer Size

This setting determines the size of the multicast socket send/receive buffers. Its value defaults to 64 KB.

You also may need to tune the behavior of your OS before multicast communication for the cluster is functioning properly. In certain situations, your cluster setup may be devastated by "multicast storms," whereby servers within the cluster are unable to process incoming messages fast enough, thereby leading to a large increase in network traffic. Here, you may need to modify the behavior of UDP socket buffers at the OS level.

14.8.2 Peer-to-Peer Communication

WebLogic supports IP-based socket communication when accessing remote objects deployed on another machine, and for replicating HTTP session state and stateful session EJB instances from a primary server to a secondary server. All remote RMI communication also occurs using sockets. For optimal performance in a cluster, be sure to follow the guidelines described in Chapter 15.

14.8.3 Administration Server Considerations

The Administration Server hosts the configuration for the entire domain, including all clusters that belong to the domain. For this reason, the Administration Server should be accessible to all Managed Servers within the domain. In addition, it should not participate in any of the clusters. A WebLogic cluster usually maps to an application tier and hosts a number of application components and services. The Administration Server should not be dedicated to application-specific tasks, but should rather focus on completing domain administration tasks.

Remember to exclude from the cluster-wide DNS name the listen address and port that is assigned to the Administration Server. For security reasons, you also may wish to place the Administration Server behind a DMZ.