Database Server Failover | Maximizing Performance and Scalability with IBM WebSphere

I've said several times throughout this book that databases and WebSphere are almost synonymous with each other in an online J2EE environment. For the most part, placing a user -oriented application on the Internet requires some form of backend database. If not only to provide tracking or logging services, databases are pivotal to a WebSphere-based platform. For that reason, in this section, I'll focus on several ways to implement or, more specifically , support database failover within WebSphere applications.

You've already focused on the performance of database interconnection with WebSphere. The other side of the equation is failover. It's important to realize that sometimes performance and failover can cancel each other out or, at the very least, impact each other.

The following sections cover these areas of failover support:

WebSphere 4 repository database failover
User application database failover support
Session database failover support
Considerations when using Oracle 9i Real Application Clusters (RAC) and Oracle Parallel Server (OPS)
Considerations when using other databases that support clustering and failover

You'll first look at WebSphere 4 and its repository failover support.

WebSphere 4 Administrative Repository Considerations

As discussed in earlier chapters of this book, WebSphere 4 maintains a repository on a relational database. The repository itself stores runtime and configuration information about WebSphere application servers and or domains. Configurations for server groups and other associated configuration areas within WebSphere are stored and maintained within the WebSphere 4 repositories.

However, it's important to note that the repository is managed by the WebSphere 4 administrative server. This administration server maintains and supports a lot of the functionality that the WebSphere 5 deployment manager and node agents support, such as Java Naming and Directory Interface (JNDI) failover and support, centralized application server runtime support, and so forth.

In Chapter 7, I discussed WebSphere 4 administration server failover but focused on the specifics of the administration server runtime failover. Because there are elements of the administration server that rely on the database during database failover, the specifics of repository failover have been left for this section.

Consider for a moment what happens to the repository during a failover event. The answer to this question depends on the state and status of your application environment. If, for example, you have two WebSphere server nodes running, with a handful of application servers and clones running in each node, and your repository became unavailable, nothing would happen. The repository is only critical during WebSphere application server failover, startup, shutdown, or configuration changes. Most of the configuration that you set up via the administration console or the WebSphere Control Program (WSCP) command line tools can only take effect after restarting an application server.

If your repository goes down for a period of a few minutes, and there's no need for your applications or WebSphere itself to query information about its constituent components , then the impact to a repository failover (or downtime) is minimal. Therefore, configuring or tuning the availability of the repository during a failover involves minimal effort.

If you're using a database that supports connection time failover (such as Oracle), then you can configure your administrative server Java Database Connectivity (JDBC) settings in your <WAS_HOME>/bin/admin.config file to support JDBC thin-driver connection time failover. This will provide your administrative server with the ability to connect to alternative repositories if you're running static or standby database servers where data replication is manual (via loading of redo and archive logs), semimanual (via data replication), or multinode configurations in an active-active database configuration.

If you're running an active-standby configuration, your JDBC Uniform Resource Locator (URL) will be already pointing to the "cluster" address. In most cases, you'll configure, with your database cluster (active-standby), a Domain Name System (DNS) entry that's the IP address takeover host name for the cluster.

That is, if you have two servers in an active-standby configuration ”one with IP address 10.0.0.1 and the other with 10.0.0.2, you'll also then have another IP address such as 192.168.1.1, which refers to the "cluster" address. Through the clusterware configuration that supports and manages the IP address takeover configuration, you configure a mechanism to facilitate the takeover and reactivation of the 192.168.1.1 address as nodes failover back and forth.

The cluster DNS name, which may be something such as yourDBCluster.mydomain.com , will refer to 192.168.1.1 in the zone file. At one point, during initial booting, the host name of 10.0.0.1 will have been configured on another network interface, 192.168.1.1, which refers to yourDBCluster.mydomain.com . If 10.0.0.1 fails, the clusterware takes over the IP address of 192.168.1.1 ( yourDBCluster.mydomain.com ) and tells 10.0.0.2 to activate it on one of its configured network interface cards. In this way, the host entry of yourDBCluster.mydomain.com is pointing to the same IP address the whole time, meaning that you can configure your JDBC URL settings for your repository to be yourDBCluster.mydomain.com .

As the database nodes failover back and forth, your repository still connects to what it thinks is the same node each time. As long as whatever mechanism you use in your database availability model supports solid and up-to-the-millisecond data synchronization, you won't have a problem. If your database replication or failover model is a little less diligent with synchronizing data, you may have a problem if you failover to stale data.

The repository updates the data store from time to time as nodes, application servers, server group , and so forth start up and shut down (enter and leave WebSphere clusters). You can probably appreciate what kind of mess would occur if you had recently fired up a WebSphere server that updated the shared repository between all the nodes in the WebSphere 4 domain, and, shortly after, the repository database had to failover to a state version of the repository that didn't have the recently booted WebSphere node in its repository data store. As soon as another WebSphere node attempted to join the cluster (or any constituent part of any of the existing nodes), the dirty or stale information in the repository would cause problems with your cluster's host maps and synchronization.

So, as you can see, maintaining the repository doesn't require a complicated failover model as long as data is always synchronized. Failover times of up to several minutes can usually be handled by most WebSphere environments with the only exceptions being changes (configuration or state) to the components within the WebSphere domain.

User Application Data Sources and Databases

Applications that use databases to store data using JDBC are well supported by WebSphere when it comes to database failovers. The JDBC Application Programming Interface (API) in both WebSphere 4 and 5 supports a range of exception messages that WebSphere as well as the application itself can use to perform certain events.

What am I talking about when I mention application databases ? Essentially , this is where your application is binding itself to a data source (in this example, a JDBC data source) that in turn pools connections to a relational database (such as DB2, SQL Server, Oracle, and so on).

Failover of these databases is a situation that can cause great headaches for both systems managers as well as application architects . So, what can happen? Well, consider a standard query to a database. The sequence of events is something like this:

Application code usually performs a JNDI context lookup to get the JDBC data source binding.
Once the application code has the reference to the data source, it'll request a connection from the connection pool.
Once the connection object has been obtained, the application code will then perform some sort of query via Container Managed Persistence (CMP), Bean Managed Persistence (BMP), Data Access Objects (DAO), or straight JDBC calls.
Once the result set has been returned, the application code, or a factory class that supports the opening and closing of connections, closes the connection to the database pooled connection object, and the pooled connection object returns to the pool manager's available connection list.

Note
This is an example of a connection to a database. There are other ways you can achieve this, but this example uses this simplistic yet legitimate approach.

At any point in these four steps, a database failover could occur; the result is different for each step and needs to be handled differently. You'll now go through each step and look at what can happen and what to do in the event of a failure.

Step 1: Application Binding

This step doesn't involve the database at all. In fact, it relies on the JNDI construct being available that's managed and supported by the administration servers in WebSphere 4 and the node agent or application servers in WebSphere 5.

Always bootstrap against the administrative server in WebSphere 4 for JNDI failover and application servers in WebSphere 5. Your JNDI lookups can then support failover.

Step 2: Pooled Connection Request

Requesting a connection to the database from the connection manager will perform satisfactorily each time within an application given that the request for a connection from the pool manager will result in a query to the local application server data source provider. As each application server instance obtains its own data source (and connection pool manager), if the pool has failed, then the application has failed. The resulting recovery steps in this process are handled in a different area within WebSphere.

If a query is performed (see the next section) via a pooled connection and the connection fails because of a downed database, then at this point one of two things can happen: First, if the application JDBC pool manager is interfacing with an active-standby database, then the pool manager's connection timeout settings will affect the outcome of what may happen. Second, if the connection timeout settings are set too low and the cluster fail takes longer than the timeout settings, then the connection will fail and either a StateConnectionException or a ConnectionWaitTimeoutException will be thrown.

Step 3: Query Performed

Once the query is performed, the data source has obtained the connection to the database and the query is made to the database. If the database suffers a failure in midflight of the query, again, depending on your database configuration and architecture, your query may fail, or it may be retried. Best practices suggest that your application query and capturing your database exceptions should be managed within a single method.

Developers should be creating their application code so that the calling method that pushes the query to the database also captures any resulting exceptions. If the resulting exception is either a StateConnectionException or a ConnectionWaitTimeoutException , then the application code should be able to handle the situation by retrying the client requests the number of times defined by the application.

If the number of retries defined in the application code is exceeded, then the application should give up and assume there's a failure in the failover process ( assuming you've coded your timeouts appropriately with your database failover period). Listing 8-1 shows an example of this implementation.

Listing 8-1: Handling Database Errors in Application Code

 int retryMax = 3; int waitValue = 5000; Public void someConnectionPool () {      // assume all appropriate java classes are loaded      try {          // assumes that the JNDI data source binding has already been made          conn = datasource.getConnection();          statement = conn.createStatement();          statement.execute("select * from my_table where user_name = 'adam'");          retry = false;      } catch (staleConnectionException s) {          // commence management of error          if (retries < retryMax) {              retry = true;              retries++;              sleep(waitValue);          } else {              retry = false;          }      } catch (SQLException se) {          // manage general SQL exception types here      } finally {          // close of connections and statements here      }      } while (retry); }

As you can see from Listing 8-1, some fairly simple logic implements some robustness in the application code. Although this doesn't necessarily support the failover of the database, it does support the ability to insulate applications from failover delays. Essentially, the application could throw an error message to the customer, asking the customer to retry his request in a few moments ”by which time you hope that the database failover has taken place.

If you're using transactions within your application code, you need to be aware of the additional overhead of performing queries and updates to a database that may fail at some point because of a failover situation or, worse , a total database environment failure. You can implement transaction control using standard Java Transaction API (JTA) or similar Java/J2EE transaction management functionality, or your developers can develop transaction management using custom-built transaction managers.

If you're using transactions within your environment, you'll need to ensure that if a situation arises where after a number of retries to a database, a request or update fails and the entire transaction needs to be rolled back. The development of this application code is beyond the scope of this book, but it's something you should consider.

If your applications are using CMP, then the EJB container will manage the retrying of the queries to the data source without the application being involved.

This discussion so far has only focused on databases that are active-standby where there's a failover period involved. If your database environment is something that's multiheaded or active-active, then you can further insulate your applications from outages. As you've seen in other chapters, an active-active database is one where there are two or more nodes, actively participating and serving in database queries. This is different from a load-balanced environment in that the load distribution is handled by the database cluster software rather than a hardware or software appliance distributing requests to multiple nodes (as in a Web server farm).

In this type of database configuration (which you'll see in more detail later in the chapter), you'll still be faced with StaleConnection and ConnectionWaitTimeout exceptions if all nodes are down. However, with an active-active database environment, you'll always have more than one node actively running and servicing requests. In most cases, the distribution mechanism is handled by the JDBC driver with specific database vendor settings that help distribute load accordingly .

In summary, follow the same rules as discussed for Listing 8-1, but be aware that your environment availability will be higher and more robust than it'd be with an active-standby database cluster.

Step 4: Closing the Connection

Closing the connection is similar to the second step where the application processes and connection pool manager are operating almost within the same space (definitely the same JVM). When the application code requests that statements and connections are closed, this doesn't affect or involve the database at all, but it helps to release unused objects from memory.

Session Database

The session database is a key part of your overall WebSphere availability strategy. This service within WebSphere ties into the HTTP plug-in mechanism discussed at length in this book. The session database essentially backs up the session manager by persisting the session object (and its contents) to a database. This provides the ability for the user session to not be lost during a failover scenario (see Chapters 6, 7, and 9 for more discussion on session management and persistence).

You'll now look at the implementations and configuration options that can help improve availability of your user sessions during database failover scenarios.

In Chapter 7 when I discussed configuring and tuning of the session persistence process, you learned about the three forms of session persistence that come standard with WebSphere 4 and 5:

Time-based persistence
End of servlet service method
Manually via the sync() method (an IBM extension)

Essentially, you need to choose from two options when implementing your persistence of sessions. Do you want to tune your environment to be potentially higher performing (a lower impact on your database) and less supportive of failovers or more supportive of failovers yet have a higher impact on your database because of more persistence writes ? Of course, you can always be sure to model your database so that it supports the high-persistence model (highly synchronized), in which case you're already aware of the managed load that'll be put onto your database.

WebSphere 4 and 5 both differ in terms of their options for session persistence. Because this chapter focuses on failover of external systems, you'll only see the persistence to the database option here. Please refer to Chapter 7 for a discussion about memory-to-memory and in-memory persistence options available in WebSphere 4 and 5.

So what is it that you're trying to achieve Simple ”if you've gone to the trouble and expense of implementing a high-end WebSphere environment and are using session management, you'll want to be able to recover, as transparently as possible, from situations where an application server has failed and the user session needs to be available from another application server clone or instance.

Figure 7-11 from the previous chapter highlights a situation where a user, User 1, is logging into a WebSphere environment. The session is managed by Server A. Figure 7-12 shows the same diagram after a failover scenario has occurred with the application servers. All user sessions are now available from the remaining Server B node. However, if the database server is what fails rather than an application server, the situation is slightly different.

If you have a single database server, you'll find that your session will be lost if you can't failover to another database node that has access to the user session tables. For this reason, my recommendation is that it's nearly pointless to have multiple WebSphere application servers and only have a single database server if the reason for having multiple application servers is redundancy or high availability. If the database server goes down, all the application servers in the world won't keep you up and running!

Then again, if your J2EE/Java-based application that happens to be running on the multiple WebSphere application servers isn't overly database-centric, then there may be a fair case for there not to be a clustered database server environment (active-standby or active-active). A failover within an active-standby database cluster environment is, like application databases, easy to recover from. The session persistence manager is supported via the Web container construct within WebSphere.

A nice feature of the WebSphere session manager capability is that when you configure the session manager to use a persistent model, you have to specify a data source. This data source is the same type of JDBC data source that your normal WebSphere applications use to communicate to their database. Therefore, the settings associated with WebSphere application data source usage can be applied to the session manager data source.

Remember, you want to ensure that if you have a failover configuration (active-standby), then you'll need your connection timeout setting to be longer than that of your tested failover period; otherwise , your session manager will timeout prior to the recovered or newly failed-over node becomes available.

If your database is an active-active architecture, the JDBC settings are no different except for the format of the URL (see the next section). In the extended JDBC URL format, you can support thick-based JDBC drivers that supply time-out and failover connection details in the JDBC URL. I'll discuss this in more detail in the next section.

In summary, if you require a high-availability WebSphere application, my recommendation is to consider what and how frequently the persistence of session objects is required. Remember, large HTTP sessions are expensive to persist to the database. As discussed in Chapters 4, 5, and 6, try to keep your HTTP session objects per user less than 8 kilobytes (KB), with a preference to maintain them at under 4KB if you're looking to utilize session persistence for session failover. Anything greater than 8KB starts to become increasingly intensive to persist to a database. You can take the blanket approach by persisting all session information at the end of each servlet method call, or you can persist session information at a defined period (for example, every 60 seconds).

There's a final approach, which is either to manually persist the session data using the sync() method call (which is an IBM extension) at the end of key session updates (for example, via a custom-built session store class) or to store information onto a database manually and avoid the session manager altogether. I suggest staying away from the latter approach unless there's some limiting factor why you can't use the WebSphere session manager. Although implementing your own session manager is possible, it does introduce a whole level of complexity that WebSphere itself does a good job doing.

Finally, by using the sync() method, persisting data at the end of key functions and methods allows you to persist information at a more granular level than at the end of each servlet call. Remember that a servlet method call may entail a whole range of backend EJB-based transactions, each of which may build up return objects that will ultimately be stored in the user session. There may be a design consideration for your application platform where the cost of performing these backend transactions warrants the need for you to be ultra -sensitive to the cause and force the persistence of session information at key strategic points of a large transaction.

Considerations When Using Oracle 8 i OPS, 9 i OPS,and 9 i RAC

As I've discussed already in this book, Oracle is one of the popular database platforms being used globally by WebSphere platforms because of its flexibility, scalability, and performance capabilities. OPS and RAC are both high-end, highly scalable relational database platform products from Oracle. Both provide active-active configurations, with Oracle 9 i RAC being the flagship of Oracle's highly available database platform technologies.

I've discussed OPS and RAC in earlier chapters in terms of general implementation considerations. In this section, I'll give you an overview of how to ensure that your WebSphere applications are insulated from failover scenarios with both products.

As you've seen, WebSphere supports JDBC access via type 2 and type 4 drivers when used with Oracle. The key for ensuring that applications aren't impacted by Oracle failover situations and for taking advantage of the advanced features of Oracle RAC is that you can use a number of JDBC driver and URL settings.

With Oracle technologies, there are two types of high-availability feature you can use. The first is Oracle Connection Time Failover (CTF). CTF is a simple JDBC driver setting that allows you to configure, in their tnsnames.ora or in their JDBC URL, values that provide failover capabilities during connection. The second option is Transparent Application Failover (TAF). This is a more advanced option and not all WebSphere implementations and models support this. This form of capability requires the use of an Oracle Call Interface (OCI) or thick-based JDBC driver (type 2).

Oracle Connect Time Failover

How this works is that a global service name is created for your Oracle database. Within the scope of the global service name for your OPS and or RAC database, you configure the various database nodes that participate within the cluster.

In this section, you'll look at a sample tnsnames.ora file first and then at an example of an extended JDBC URL that provides CTF. Listing 8-2 shows an example tnsnames.ora file that has the CTF extensions in it. DBServerA and DBServerB are the two nodes in this OPS or RAC cluster.

Listing 8-2: Oracle Transparent Network Substrate (TNS) Names Configuration File for Server Failover

 MyDBCluster.mydomain.com=     {description=My DB Cluster          (load_balance=off)          (failover=on)          (address=              (protocol=tcp)              (host=DBServerA)              (port=1521))          (address=          (protocol=tcp)              (host=DBServerB)              (port=1521))          (connect_data=              (service_name=MyDBCluster.mydomain.com))))

Note

The syntax of the tnsnames.ora file is a standard Oracle implementation that's used to support and find listeners within your environment. The configuration and syntax details of this file are beyond the scope of this book. Please refer to your Oracle administration and installation guides for more information on the syntax of these files.

The key lines to this tnsnames.ora file are the load_balance and failover directives. As you can see, this example shows that load balancing is off and failover is on. You can have both active by setting them both to on. If your environment is tuned to service requests in a true active-active configuration, then ensure that the failover setting is on.

If you have a topology where you may have two WebSphere "channels," you may consider having each channel with the tnsnames.ora file configured differently, with the load_balance directive turned off. That is, channel 2 would have DBServerB first, and channel 1 would have DBServerA first. With the load-balance directive off, the driver picks the first server from the list to which it can connect.

The load balancing isn't true load balancing. Instead, it's a randomness of connectivity to one of the hosts . This is good ”what you don't want to have is a situation where all your applications bootstrap and connect to the first working database server. In this case, DBServerA would be taking all the requests, and the load differential would be skewed between the database server nodes.

You can use the same configuration in a JDBC URL. The previous line could be placed in a JDBC URL as shown here:

 JDBCDriver=" jdbc:oracle:thin:@ {description=My DB Cluster (load_balance=off)    (failover=on)   \(address= (protocol=tcp) (host=DBServerA) (port=1521))    (address= (protocol=tcp)   (host=DBServerB) (port=1521)) (connect_data=    (service_name=MyDBCluster.mydomain.com))))"

In implementation, this URL would typically be one long line. As you can see, it's quite an ugly-looking string. Configuring this in your JDBC data source provider is a little sore on the eye, but the end result is robust failover support.

How this JDBC configuration works is this: If DBServerA fails, the JDBC driver tries to connect to the DBServerB. The driver will keep trying each server until one becomes available, should both servers be unavailable.

Oracle Transparent Application Failover

Oracle TAF is a more advanced version of CTF. Essentially, this capability provides a completely transparent failover and connection layer to application and WebSphere databases. Using a thick-based JDBC driver implementation, TAF provides the same functionality that CTF provides, but it supports active connections rather than new connections (at startup or through a failover).

Similar to the CTF implementation, TAF is implemented through the tnsnames.ora configuration file on each WebSphere host. Much of the syntax is the same, except for some extensions that provide more advanced redundancy and robustness in the application connections to the database.

Listing 8-3 shows a TAF-based tnsnames.ora file, which takes the same configuration file from Listing 8-2 and adds directives that are the TAF-specific settings. These are highlighted in bold.

Listing 8-3: Oracle TNS Names Configuration File for Transparent Application Failover

 MyDBCluster.mydomain.com=     {description=My DB Cluster          (load_balance=on)          (failover=on)          (address=              (protocol=tcp)              (host=DBServerA)              (port=1521))          (address=              (protocol=tcp)              (host=DBServerB)              (port=1521))          (connect_data=              (service_name=MyDBCluster.mydomain.com)  (failover_mode=   (type=session)   (method=basic)   (retries=5)   (delay=5))))

These additional lines provide directives to the JDBC driver to direct it to load itself up in a specific manner. That is, in Listing 8-3, the main lines of importance are the type and method lines.

The type defines the type of failure required. For this directive, there are three options: Session, Select, and None. They can be summarized as follows :

Session: The Session type provides a high level of failover support. Specifically, this type fails the application's pooled connections to the other host. There's no query failover supported within this option.
Select: The Select type is more thorough than the type Session. In this option, all queries and open cursors are maintained across failover, and generally no impact is found noticed with this option.
None: The None type, as it suggests, doesn't attempt to failover any active queries or cursors.

The method directive defines how the failover should occur. There are two settings available for this directive: Basic and Preconnect. They can be summarized as follows:

Basic: The Basic method provides the capability for the failover connection to the failover node to occur at the time it's required ”a failover. There's no overhead in maintaining this form of failover method.
Preconnect: The Preconnect method supports an advanced failover capability that sees that all connections established on one host (for example, DBServerA) are opened and maintained on the second host (for example, DBServerB). This, as you can appreciate, introduces overhead on the secondary server but provides an ultra-transparent failover configuration for your applications.

From these details, if you're looking for an ultimate failover design, the best options to choose are the Select type and Preconnect method. There have been reported problems in various Internet-based discussion groups on the Select type with WebSphere. For this reason, ensure that if you want to use this type, test failover thoroughly during stress and volume testing. Alternatively, you can opt for the Session type, which will guard against most failures and, in this case, have the application code rerequest the query.

If you use a Session type and a failover occurs with an inflight query, your applications will be thrown a StaleConnection exception. As discussed earlier in this chapter, implement application logic to retry the query or request, or if your applications are using CMPs, the container will rerequest the query automatically.

Using Other Active-Standby or IP Takeover Clustering Systems

You can summarize all database failover events and resulting environment states in three parts :

Complete and total environment failure after a database server/cluster fails
Semimanaged, semitransparent failover where inflight transactions will most likely need to be retried
Fully transparent database failover to applications

The first possible scenario is something that'll likely only occur when there aren't remaining nodes available in a cluster or when there's only a single database server operating. The second scenario will occur in an active-standby or hot-standby database configuration where there may be an unusually long period of failover that causes, or requires, the applications to retry their queries several times. The third scenario is where something such as an Oracle 9i RAC or DB2 Enterprise Edition database (with Parallel Sysplex) is being used to transparently serve requests between multiple hosts.

Of all the databases supported by IBM WebSphere, each of them supports at least the second scenario where you can implement basic clustering to support failover with hot-data replication (via a shared or mirrored storage system).

In this form of clustering, the databases are clustered and are exposed to your network and WebSphere environments via a clustered IP address or host name. As discussed in earlier sections, typically this form of IP-takeover clustered solution simply shuts down and starts up network interfaces between one or more nodes, which makes the database appear as though it's always available, with slight delays or pauses in query services.

If you're concerned about high availability and must have some form of always-available database platform, be sure to implement your database environment with, at the very least, IP-takeover clustered solutions (for example, active-standby). If your WebSphere-based applications are mission critical and just can't go down (or the costs of them going down is hugely expensive), then consider one of the active-active database platforms available such as Oracle 9i RAC and DB2 Enterprise Edition database (with Parallel Sysplex). Each of these databases is available on all supported WebSphere operating systems.