Optimization and Tuning Checklist

I'm a strong believer in checklists. It's amazing how many times things don't go according to plan because simple things don't get completed or done correctly. That's why a checklist is a good thing. Checklists should be living artifacts that get updated before and after each change or deployment or integration that you perform with WebSphere-based applications.

Given the myriad of configuration options and combinations of settings available within WebSphere, changes and deployments can be complex activities. This is especially evident when you're dealing with production systems and deployments are being carried out live ”in situ ”while not affecting existing traffic!

Although this chapter doesn't go into details about deployment checklists, it provides a performance tuning checklist. I use a checklist similar to the one presented in this chapter each time I deploy a new WebSphere system or new application. It allows you to check off pieces that need to be looked at, tuned , or optimized based on the application being deployed.

Many of these settings and recommendations are based on best practices, proven baselines, and battle- tested platforms. Most of the recommendations aren't WebSphere specific; however, they, when combined with a WebSphere-based application environment, do provide a fair degree of performance and optimization improvements.

Your mileage may vary depending on a range of factors regarding your hardware (for example, based on your bus, network controllers, disk and disk technologies, and memory types). However, for the most part, you'll find these recommendations invaluable for your WebSphere-based application environments.

Operating Systems

Given that WebSphere will always sit on top of an operating system of some sort , the operating system therefore forms a crucial part of the WebSphere environment. Further, WebSphere can operate on many different operating systems, including anything from a Pocket PC 2003 operating system on Personal Digital Assistant (PDA) to a zSeries IBM mainframe.

However, you'll focus on some of the more mainstream operating systems on which WebSphere is most commonly deployed:

• Windows NT, 2000, XP, and 2003

• Linux

• Solaris

• AIX

Many of these points are non-WebSphere specific in that you can apply them in any instance for distributed-type computing applications. However, there are a handful of them that are known and proven to work well with WebSphere.

Windows NT, 2000, XP, and 2003

Microsoft's server platform operating systems ”Windows NT, Windows 2000, Windows XP, and Windows 2003 (supported as of WebSphere 5.02) ”are popular for smaller- and medium- sized environments. Although the Windows server market has been coming under fire from Linux over the recent 12 “24 months, the well-known Graphical User Interface (GUI)-based operating system can provide a good solid platform choice for use with WebSphere.

Most of the tuning aspects of the Windows server operating systems can be applied across all these server types. You'll look at these in the following sections and then see the specific Windows server tuning and optimization tips for WebSphere.

General Windows Server Operating System

Windows systems use a registry-based tool for centralized configuration information. Using the Registry Editor tool, you'll be able to change the settings within the registry as you see fit.

Caution  

The Windows registry is the centralized configuration area for your Windows server. As the administrator user, you can change any setting within the server. Be careful about what you change, however, and always make backups of the registry before you commit any changes.

Maximum TCP Port Number Assignment

This setting allows you to bind WebSphere to ports beyond the default range. This additional registry setting will help with larger WebSphere installations on a Windows-based server where you may have a need for many ports.

To make the changes, follow these steps:

1. Load your Registry Editor (in other words, regedit ).

2. Go to the registry directive of \ HKEY_LOCAL_MACHINES\SYSTEM\ CurrentControlSet\Services\TCPIP\Parameters .

3. Most likely, the registry entry of MaxUserPort won't be present. Create a new entry, and set the value to be greater than 32,768 but less than 65,536.

4. Restart the system.

This setting is essentially allowing WebSphere's dynamically allocated ports to go beyond the default number. Although this may not be a problem for all installations, Windows-based WebSphere implementations that are large and have many connections from a Web tier or many internal application servers may receive a performance boost from this.

The complication of this sitting not being high enough is that application instances or network connections attempting to bind to a higher port or a port that's not available will result in errors.

This setting can be tied to end user Web browsers and Hypertext Transfer Protocol (HTTP) keepalive configurations. The more your environment and ultimately your end users' browsers can take advantage of keepalives , the more efficient the use of system network points associated with this setting will be.

TCPTimedWait Delay

The TCPTimedWaitDelay value within Windows NT, 2000, and XP defines the time between when a connection has ceased (closed) and when the operating system can relinquish the connection handle to other resources (such as new connections).

The telltale sign of this setting not being correctly tuned for your WebSphere application environment is when running the netstat command and seeing many TIME_WAIT entries in the STATE column of a netstat command's output.

The TIME_WAIT state is a period between when the Transmission Control Protocol/Internet Protocol (TCP/IP) stack has closed a connection between a client and server but hasn't released the connection itself. This is somewhat analogous to file handlers ”if you delete a file that's opened by an application, the file isn't really gone if the application has an open handle to it. The application needs to be cycled or forced to drop the handle in order for the file to be released.

You should change this setting if you've noticed that new connections between a client and the WebSphere server (for example, thick Java clients , lots of external socket connections, and so on) were sluggish and operating at a low throughput.

This doesn't always mean that the WebSphere application server or the operating system are operating or performing badly , and you may even find that the transactions, once established, operate at normal speed. It's a stress versus volume issue ”poor volume capability versus good stress capability.

Essentially, you need to decrease the TIME_WAIT interval time from the default of 240 seconds (which is four minutes) to a lower amount.

To make changes to this setting, follow these steps:

1. Load your Registry Editor (for example, regedit ).

2. Go to the registry directive of \HKEY_LOCAL_MACHINES\SYSTEM\ CurrentControlSet\Services\TCPIP\Parameters .

3. Similar to MaxUserPort, most likely the registry entry of TCPTimedWaitDelay won't be present. Create a new entry, and set the value to be between 30 seconds and 60 seconds. The value needs to be entered in hexadecimal format with 30 seconds equaling 0x0000001e and 60 seconds equaling 0x0000003c.

4. Restart the system.

After you've restarted the system, monitor the output of netstat and track (using some form of graphing or plotting tool) the frequency of TIME_WAIT values. You should see the amount of TIME_WAIT instances drop. You should, however, see a noticeable increase in new connection transaction rate to your WebSphere server.

Linux

Linux is undoubtedly the fastest growing operating system on the market. Since its inception in the early 1990s, Linux's acceptance and installed user base has been increasing almost exponentially.

IBM embraced Linux several years ago and invested significant amounts of funding and resources in the emerging operating system. Now, with Linux ported to more hardware platforms than any other operating system, it's set for a long history within the Information Technology (IT) industry. It didn't take IBM long until it ported the WebSphere platform to Linux.

For what it's worth, I'm a Linux enthusiast, so I take great interest in that I can install WebSphere on an old desktop that may be laying around my house and, within an hour or so, have Linux installed and WebSphere applications up and running.

You'll now look at some of the settings available for a high-performing WebSphere environment running under Linux.

Note  

These tuning and optimization recommendations are correct for 2.2. x to 2.4. x Linux kernels . With the rapid pace of Linux kernel development, please check the errata and README instructions in your kernel source tree to determine whether these settings are still required in your potentially newer version of Linux. That said, these settings should apply to newer kernel versions without any adverse performance degradation.

Linux TCP/IP Stack Optimization: Turning Off Unnecessary Features

Most of the time, your WebSphere application will operate fairly network- intensive applications. Some WebSphere installations will notice or deem this setting as a priority more so than others, but for the most part it's prudent to look at optimizing your network settings as much as possible.

Two features of the Linux TCP/IP stack, although "cool" features, can impede performance in a WebSphere environment.

Caution  

The following settings will need to be updated in your startup scripts to preserve the new settings after a reboot.

TCP Selective Acknowledgment Disabling

The first setting to deactivate is the TCP Selective Acknowledgments setting. This feature alters the way in which select acknowledgment messages are handled for TCP communications, which is a feature that's not required for WebSphere optimization or functionality.

To deactivate the feature, run the following command as root on your Linux-based WebSphere application server:

 [root@somenode: /home]  echo 0 >/proc/sys/net/ipv4tcp_sack 

A powerful feature of the Linux kernel is that you can modify many of its settings and features dynamically by directing commands to the proc file system.

Note  

The proc file system on the Linux operating system (and to a degree, Unix in general) is a file system view of the active processes and elements in memory. Linux extends this further by loading kernel parameters into this memory space and allowing elements and options to be changed on the fly.

TCP Time Stamp Disabling

Another on-by-default feature of Linux is TCP time stamping. This TCP feature is primarily meant for, but not limited to, long-distance routes where Round Trip Time (RTT) and other similar algorithms are calculated to aid in proper routing and TCP protocol framing and transmissions.

Unless your Linux-based WebSphere application environment is doubling as a host-based router (!), then you can safely turn this feature off.

To deactivate the feature, run the following command as the root on your Linux-based WebSphere application server:

 [root@somenode: /home]  echo 0 >/proc/sys/net/ipv4/tcp_timestamps 

Problems with TCP Keepalive

The Linux kernel by default has a TCP keepalive value of two hours, which, on a busy WebSphere-based system, can cause problems with long-held TCP sockets.

This can be a feature or near requirement on certain systems, but for WebSphere-based applications (or any J2EE application server for that matter) that tend to exhibit smaller transactions (in other words, higher transaction rate, small transaction characteristic), there's little or no need to manage keepalives beyond a few minutes. Essentially, there are other ways in which this can be managed at the application level.

To deactivate the feature, follow these steps:

1. As root on your Linux-based WebSphere application server(s), you'll need to edit the kernel source and recompile it.

   Note  

   Recompiling the kernel is beyond the scope of this book; however, a good Internet resource for kernel recompilation is http://www.kernel.org .

2. Using a text editor (for example, vi, emacs, joe, and so on), open the file to edit:

    $SOURCE_BASE/include/net/tcp.h 

3. Search for this line:

    define TCP_KEEPALIVE_TIME (120*60*HZ) 

4. Change the value to this:

    define TCP_KEEPALIVE_TIME (5*60*HZ) 

5. Save the changes, and exit.

6. Recompile, deploy, and reboot your server(s).

Increasing Socket Buffer Memory Allocation

The socket buffer is the context that's used for all new TCP-based connections within WebSphere. Like most things, the more memory you can allocate to the buffers, the better performing your WebSphere-based applications should be.

There are two settings you need to change to optimize the socket buffer, or there are four to change if your application environment is making a lot of outbound connections to other systems.

As root on your Linux based WebSphere application server(s), run the following command:

 [root@somenode: /home]  echo 262143 > /proc/sys/net/core/rmem_max [root@somenode: /home]  echo 262143 > /proc/sys/net/core/rmem_default 

This setting increases the buffer space enormously from the default values. Be aware that this buffer increase needs to be calculated during capacity modeling. You'll always want to make sure you have sufficient memory available and allocated to these types of buffer spaces.

If your WebSphere applications are performing a lot of outbound connections, issue the following commands as root on your Linux-based WebSphere application server:

 [root@somenode: /home]  echo 262143 > /proc/sys/net/core/wmem_max [root@somenode: /home]  echo 262143 > /proc/sys/net/core/wmem_default 

Given that the settings are instantly active, you should be able to perform testing or profiling again.

Increasing the Amount of Open Files

This setting is of most use to those sites that have many Web presentation components. That is, for a UI with many components within it, such as GIF and JPG image files, this setting will affect the volume of traffic your WebSphere system can support.

An example symptom of what this fix will achieve is where you have a Web-based interface and images are missing within the output. Although this isn't always caused by an open file limit issue, chances are that it will be!

This setting really just allows your WebSphere-based applications to access and reference more files. If you have a larger site or a high-volume site, this setting is a must for your Linux server.

Depending on which versions of the kernel you have, there are two approaches to this setting.

Pre-Linux 2.2

Pre-Linux 2.2 such as 2.1 and even possibly 2.0 systems (which aren't always necessarily supported by WebSphere) provide the ability to change the number of open files on the system via configuring the proc file system.

To make the updates, run the following command as the root on your Linux-based WebSphere application server:

 [root@somenode: /home]  echo 8192 > /proc/sys/kernel/file-max 

This sets the value to be 8,192 open files available to your WebSphere server.

Keep in mind that the more files you have open, the more buffer space (memory) is used. A rough guide to how much you should set this to is 4,096 for every 64MB of memory.

You'll also need to increase the number of inode handlers that are possible. Inodes in Unix are a reference or handle to a file. Depending on how you configure your file system, your files on your disk may require several inodes. For this reason, I recommend setting your maximum inode value to four times that of your open files.

To change this setting, run the following command as root on your Linux-based WebSphere application server:

 [root@somenode: /home]  echo 32768 > /proc/sys/kernel/inode-max 

The value of 32,768 is four times that of the value 8,192 from the previous example.

Linux 2.2+

Changes to how the proc file system was structured were made between Linux kernel versions 2.0/2.1 and 2.2+. As such, the proc elements that need to be configured and changed are different.

The same setting recommendations and guidelines mentioned previously apply for Linux 2.2 and higher.

To change this setting on Linux 2.2 and higher, as root on your Linux-based WebSphere application server(s), run the following command:

 [root@somenode: /home]  echo 8192 > /proc/sys/fs/file-max [root@somenode: /home]  echo 32768 > /proc/sys/fs/inode-max 

Linux and Lightweight Processes (LWPs)

A mechanism in several Unix variants, including Linux, exists called Lightweight Processes (LWPs). LWPs are units of operational runtime that exist between user-level threads and kernel-level threads and provide a different paradigm for running applications and processes from that of more the more traditional kernel and user-level threads and processes.

LWPs can be useful for providing a more granular method of system (memory predominately) management within a Linux server. However, be cautious of getting confused when seeing process tables. LWP-based threads don't operate in the same nature as a standard process under a user-level thread does.

One of the key reasons for using LWPs is that (if you recall back to earlier discussions on context switches within a server CPU) the LWP, because of its lightweight model, can be context switched faster on and off a CPU than a standard process. For this reason, it can be advantageous for performance to use LWP libraries within your Linux or Unix server.

LWPs can be of great assistance for smaller environments that may have only a single processor. Because the LWP libraries provide a quasi-multithreaded construct for applications to operate within, you can obtain increases in WebSphere application performance on single-CPU systems through the more efficient use of system resources.

Before you use LWPs, however, be sure you understand their implementation and system architectural implications.

Sun Solaris

Sun Solaris, once known as SunOS , is Sun Microsystems' long-time Unix-like operating system, developed and available for both the SPARC and x86 platforms. Over the past 12 months, Sun has been toying with the Linux operating system. Indications from reviews, reports , and insider information are that Sun sees Solaris as an enterprise operating system while supporting Linux (previously a Red Hat variant) on lower-end systems (such as the x86).

Like Linux and AIX, there's a plethora of best practices regarding tuning the Sun Solaris operating system for various applications.

For WebSphere-based applications, again like Linux and AIX, the tuning and optimization is based mostly around the networking side of the operating system. This is primarily because of the sometimes not so obvious fact that J2EE applications are inherently network-centric. EJBs by their nature are distributed (networkable) objects, and, as such, optimizing your networking layer for both hardware and software will always give you positive gains.

In the following sections, you'll first take a look at the networking tuning that I recommend for your WebSphere-based servers. The networking optimization recommendations for Solaris aren't too different from that of the other operating systems discussed in this book.

Proper TCP resource utilization is really the key to high WebSphere performance. You could have a massively scaled Symmetric Multiprocessor (SMP)-based server with tens or even hundreds of CPUs, but if your networking settings aren't correctly tuned, WebSphere and your J2EE applications won't function in the way you want.

On smaller systems, you can get away without these settings. That is, development nodes and quite small production environments wouldn't notice the difference, in most cases, if these up coming recommendations aren't applied. However, you'll notice a performance impact if they're not correctly set for higher-end environments. If you have the luxury of a solid stress and volume testing environment, try it out for yourself.

Caution  

The following settings that use the ndd command aren't permanent and are only active for the duration that the server is up and running. To preserve them after a reboot, look at including them in your startup scripts on boot.

TCP Keepalive Settings

Similar to the Linux operating system recommendation, the TCP keepalive setting can be helpful in some networking scenarios; however, for the atypical WebSphere-based implementation, especially a fair-sized one (medium to large), keeping TCP connections open for too long in the hope that a client will continue to make requests down the same keepalive connection will exhaust the available TCP sockets.

This will also result in a memory usage problem because each additional connection consumes memory as well as CPU cycles. You should aim, as a rule, to ensure that as many CPU cycles as possible can be provided to process application and user requests, rather than unnecessary house cleaning.

You'll now look at the current TCP keepalive setting to determine what your final setting should be. First, to find out what the current setting on your Solaris box is, issue the following command:

 [root@labserver1: /home]  ndd -get /dev/tcp tcp_keepalive_interval 

A number should come back; if you haven't already changed this number, it should be set to the default value of 720,000 (seconds), or 12 minutes.

Based on your application characteristics, you should be able to calculate what the maximum window should be. The interval window should be small enough to ensure that connections are being managed efficiently yet, at the same time, not so large that your environment or your TCP stack is running out of available connections.

For WebSphere on Solaris, my recommendation is to set this value between 2.5 and 5 minutes, with the values of 150,000 and 300,000.

To make changes to this setting, as root on your Solaris-based WebSphere application server, run the following command:

 [root@labserver1: /home]  ndd -set /dev/tcp tcp_keepalive_interval 150000 

The setting at this point is made, and no reboot is required.

Tip  

After making changes such as this one, it's worthwhile to keep track of what difference the setting change makes to your environment. A simple method is to continuously count the number of established connections before and after the change over a period of time using a small script. You'll look at this in Chapter 14.

TCP TIME_WAIT interval

This setting is similar to the TIME_WAIT setting discussed for the Windows-based server configurations. Following along the same principles, the TCP TIME_WAIT interval setting relates to a situation where connection handles are maintained or, kept open, for too long a period once a connection has in fact ceased.

This affects a system because the server will start to become sluggish and poor performing while TCP connections remain open, unneeded.

If you're able to test this situation in a stress test environment, you'd find that the WebSphere application server starts to become bogged down with system-based processing tasks, and user space application tasks (when looking at vmstat , for example) will decrease rapidly while the operating system attempts to manage all the connections.

In extreme cases, you'll find that connections can be refused if the active list of connections is too high.

This setting is in fact set up by default in the WebSphere startup scripts; this only works if you run your WebSphere server as the root user (which you shouldn't). To view what the current setting is, run the following command:

 [root@labserver1: /home]  ndd -get /dev/tcp tcp_time_wait_interval 

The returning value, if it's set as default for Sun Solaris, will be 240,000 ( milliseconds ). This value is a little high; for large or high volume sites, you should look to decrease it.

Solaris lets you set the value as a minimal 30 seconds (30,000 milliseconds). Depending on your application usage and volume levels, this value should be set no longer than 90 seconds.

I recommend you start at 60 seconds and profile your environment from there. To make changes to this setting, as root on your Solaris-based WebSphere application server, run the following command:

 [root@labserver1: /home]  ndd -set /dev/tcp tcp_time_wait_interval 60000 

No reboots or restarts are required; the changes are instantly active.

FIN_WAIT_2 Problem with Solaris

With similar consequences to not correctly tuning the TCP_WAIT setting in Solaris, the FIN_WAIT_2_FLUSH_INTERVAL setting is also important.

The FIN_WAIT_2 state is a TCP state that occurs when the server running WebSphere attempts to close the connection with the client. The sequence of events for a connection closure is the server sending the client a FIN bit directive. The client is meant to respond with an ACK packet, indicating that it has received the connection termination request (in other words, the FIN packet). The client then sends a FIN packet directive to the server, which in turn sends back an ACK packet.

During the period that this is occurring, the server places the connection into a state of FIN_WAIT_2. Of course, if the network path between the server and client is overutilized or there's some performance degradation somewhere in the middle (or on either of the servers), this period of FIN_WAIT_2 can continue indefinitely.

Some operating systems such as Solaris allow you to cap this interval period.

The consequences of not having a correctly set timeout value is that the server will continue to try to maintain the handle on the nonterminated connection resource and could eventually run out of new connection handlers or cause the server to run out of memory in extreme cases. Remember, the operating system will consume system resources (CPU and memory) for each open connection. Given that these resources have finite capacity, you want to be sure that they're operating efficiently.

To view the current setting, run this command:

 [root@labserver1: /home]  ndd -get /dev/tcp tcp_fin_wait_2_flush_interval 

The default Solaris setting is 675,000 milliseconds, or approximately 11.5 minutes.

As you can appreciate, if a close connection request is attempting to take this long, there's surely a problem!

You can decrease the setting to anything you desire ; for WebSphere-based environments, my recommendation is to consider starting at 30 seconds but using nothing longer than 90 seconds.

Note that the settings are set, like other Solaris settings, in milliseconds. To make changes to this setting, as root on your Solaris-based WebSphere application server, run the following command:

 [root@labserver1: /home]  ndd -set /dev/tcp tcp_fin_wait_2_refresh_interval 60000 

No reboots or restarts are required; the changes are instantly active.

Solaris TCP Time Stamp

Identical to the Linux TCP time stamp directive, disabling this feature can provide a slight improvement of network communications for high-volume WebSphere applications. Again, be sure that you really don't need this; in more than 95 percent of the cases, you won't.

To determine the current setting, run this command:

 [root@labserver1: /home]  ndd -get /dev/tcp tcp_tstamp_always 

If the result is 1, the setting or feature is enabled. To disable it, run this command:

 [root@labserver1: /home]  ndd -set /dev/tcp tcp_tstamp_always 0 

This will take effect immediately and won't require a reboot.

Solaris Maximum TCP Buffer Size

In the WebSphere networking environment, if your platform is operating on a LAN or other form of high-speed network as opposed to over a Wide Area Network (WAN), changing this setting can improve the performance for WebSphere components talking to other fast-connected systems.

Essentially this is the maximum TCP buffer size available for data transmission. The more you can buffer into memory, the faster networking performance typically is. This value is somewhat dynamic, and WebSphere uses libraries that can change this on the fly. However, this doesn't always occur in the manner you may want.

To determine the current setting, run this command:

 [root@labserver1: /home]  ndd -get /dev/tcp tcp_max_buf 

The result will be a number between 8,192 and 1,073,741,824. The easiest way to set this directive is to match the configuration against the link speed of your network. If you're operating a 100 megabits per second (Mbps) network infrastructure, use the value of 104,857,600. For example, use this command:

 [root@labserver1: /home]  ndd -set /dev/tcp tcp_max_buf 104,857,600 

This is a setting that can have a fairly adverse affect on performance, so be sure you've tested it adequately before running this in a production environment.

Solaris TCP Connection Hash Table Setting

The TCP connection hash table is useful for large sites that have thousands or tens of thousands of connections to their WebSphere application servers. The problem with large connection environments is that not only do you require plenty of processing and memory capacity for the application runtime, but you'll also need a degree of overhead to cater for Input/Output (I/O) and networking load.

The flip side of this is that if you allocate a huge TCP connection hash table size, you'll waste valuable memory resources and CPU cycles to maintain this hash.

You set this differently than those you've seen so far. The setting is one that changes in the system file. That is, it's in the / etc/system file, which is used by the kernel to set system-based properties at boot. As such, when you change this value, you'll need to reboot the machine for the changes to take effect.

You may already have this value set; you can qualify this by opening the /etc/system file and looking for an entry that looks like this:

 set tcp:tcp_conn_hash_size=512 

The value 512 is the size of the TCP connection hash table. The setting must be a number of the power of 2. Essentially, the default of 512 will support a few thousand connections. However, if your system is going to be many thousands of connections, increase the figure appropriately.

The setting range is between 512 and 1,073,741,824. As a rough guide, start at 512 for every 2,000 connections. For example, for 10,000 connections, set the value to 2,500.

You should test this setting along with the previously discussed settings prior to implementation.

Solaris Process Semaphores

Processes and threads within Solaris use semaphores for a range of per-process reasons. Semaphores play a big part in shared memory synchronization and under Solaris. Because of the memory-intensive nature of Java, the reason why semaphore parameters may be important to tune for your environment should be obvious!

Under a WebSphere-based environment where there can be many thousands of active threads operating (on large systems), the relationship between the JVM and operating system at the process level can quickly overwhelm a system with default values for these semaphore settings.

If you're running a system-based process analyzer in conjunction with a thread analyzer within your JVM space, you'll be able to see how these two correlate and where the hotspots appear under load. Therefore, to help ensure that semaphore locking isn't causing the operating system to lag, you should consider the following settings.

The first option you should consider sets the number of operations per semaphore semop() call. This setting allows the semaphores to not be serial in nature (in other words, to process more).

The default setting for Solaris is 10; I recommend you change this to 200 to 500 depending on your system's size. Start at 200, and if you believe that isn't sufficient, increase it by 50 each time. The upper limit is 2,147,482,637. To change this setting, look for the following line in the /etc/system file:

 set semsys:seminfo_semopm 

If it doesn't exist, add it and set to the value to be as follows :

 set semsys:seminfo_semopm = 250 

The second entry sets the number of maximum number of semaphore undo entries per system process. This setting is set to 10 by default and is far too low for a WebSphere environment. You can determine if the entry is set in to the /etc/system file by looking for the entry as follows:

 set semsys:seminfo_semume 

The option may or may not have a value set, and it may be commented out. You should set this to at least 512 for midsized systems or 1,024 for larger systems. Again, use the recommended values as a baseline and then test to determine if you need to alter the settings.

Chances are that the value of 1,024 will be sufficient for most medium and large systems. Therefore, to change this setting, open the /etc/system file and either change or add the line as follows:

 set semsys:seminfo_semume = 1024 

Save the file, and you'll then need to reboot for both settings to take effect.

AIX

AIX is IBM's long-time enterprise Unix operating system that, like Sun Solaris, was its pride enterprise Unix operating system for mid-range systems (in other words, nonmainframes). AIX appears to slowly be integrating closer and closer with Linux; however, for the meantime, AIX is its own entity.

For WebSphere-based systems, AIX has a number of optional parameters that you can set and tune to optimize the performance of a WebSphere-based application environment.

You may be noticing a pattern emerging between the operating systems discussed so far. Several common optimization strategies are common between all four platforms ”those regarding the network configurations.

TCP Timewait Value

Operating the same as the TIME_WAIT settings for both Solaris and Windows, the AIX Timewait setting should be changed for WebSphere-based implementations. To recap why this is recommended, the Timewait interval helps to prevent excessive termination delays between the time when connection handles are maintained, or kept open, once a connection has in fact ceased.

With AIX, the same symptoms can occur ”connection lists can become too large and trigger any number of system resource exhaustion problems.

To identify what your setting is at, you use the AIX no command (which stands for network option ). To find out the current setting for the Timewait interval, use the following command:

 [root@aixserver1: /home]  no -d tcp_timewait 

This will return a result that will display a value. If the default value is set, the command will return 1. The value is read as 1 for every 15 seconds, whereby setting the value to 4 will increase the time wait interval to 60 seconds.

As a guide, you should use a value of 4, which equals 60 seconds, for your AIX WebSphere implementation. I suggest you shouldn't increase this value to longer than 90 seconds (a value of 6).

To set the value to 4, use the following command:

 [root@aixserver1: /home]  no -d tcp_timewait=4 

These settings take effect immediately with no reboot.

Caution  

Changing these settings doesn't mean they're permanent. You'll need to include this command in your startup scripts during boot.

Delay Acknowledgment

This setting provides the ability to delay the acknowledgment, or ACK directive, in your TCP startup and shutdown communications. Altering the setting from the default allows the ACK directives to almost "piggyback" the next packet in the transmission sequence.

By changing this value, you can gain a performance improvement with Web clients communicating to your Web tier or your WebSphere application server by decreasing the amount of overhead associated with the transaction.

Note  

To use this command, you must use it in conjunction with the delayackports directive. I'll discuss this in the next section.

There are four settings to this command:

• : No delays; normal operation

• 1 : Delays the ACK for the WebSphere or HTTP server's SYN

• 2 : Delays the ACK for the WebSphere or HTTP server's FIN

• 3 : Delays both the ACKs for the SYN and FIN directives

I suggest you use the option of 3 for the delay acknowledgment setting because this appears to achieve the best results for a WebSphere-like environment.

To determine the current setting value, use the no command with the following attributes:

 [root@aixserver1: /home]  no -d delayack 

This will return the currently set value for the delay acknowledgment setting.

To set the values, you must use the same no command, as follows:

 [root@aixserver1: /home]  no -d delayack=3 

Like I mentioned in the previous optimization recommendation, this setting isn't persistence and should be included in your system's startup scripts to ensure that it's fixed.

Delay Acknowledgment Ports

To use the delay acknowledgment directive in the previous section, you must configure the AIX networking stack to set the ports on which the delayack is valid.

The ports you choose to use with this setting on are both the Web ports to your Web containers, your HTTP server ports, and possibly the ports used by the EJB container in each of your WebSphere application servers. This ensures that you don't affect any other ports on the system, which may not benefit from changing a setting such as delay acknowledgment.

To determine what the currently configured ports are, use the no command to query the attribute of delayackports as follows:

 [root@aixserver1: /home]  no -d delayackports 

To set the ports you want to use, you can first clear the current settings by issuing the following command:

 [root@aixserver1: /home]  no -d delayackports={ } 

This wipes the existing ports from the configured list. To add ports, you must enter them between the curly brackets; you can specify up to 10, separated by columns . For example:

 [root@aixserver1: /home]  no -d delayackports={8080,8111,443} 

You've now set two ports on our WebSphere server, plus a third port that's a local HTTPS Web server.

Caution  

I don't recommend changing any other port settings other than those in WebSphere and the HTTP server. Altering other ports on the system may have an adverse effect on the operations of other processes and services (in other words, non-WebSphere processes).

AIX TCP Send and Receive Buffer

The TCP send and receive buffer attributes are simple to modify but are somewhat significant settings. They help provide more memory buffer for the TCP stack to help facilitate higher network throughputs.

The default settings are 16KB, or 16,384 bytes. In a WebSphere-based environment where network traffic is a key to overall performance, this default setting is too low. I recommend increasing this value to 64KB, or 65,536 bytes.

To check what the value is set to, use the no command to query the two send and receive buffers:

 [root@aixserver1: /home]  no -d tcp_sendspace [root@aixserver1: /home]  no -d tcp_recvspace 

The sendspace attribute refers to the output buffer, and the recvspace attribute refers to the receive input buffer.

To change these settings, use the following commands:

 [root@aixserver1: /home]  no -d tcp_sendspace=65536 [root@aixserver1: /home]  no -d tcp_recvspace=65535 

Increasing the value may have a further positive result for your WebSphere-based application; however, the larger the buffers are, the more buffer scanning the operating system needs to perform. There will be a point where this setting can be set too high and ultimately cause performance issues. I suggest you use the 64KB value; if you're not satisfied with the performance, change the setting and profile.

TCP Maximum Listen Backlog

The TCP maximum listen backlog setting is a simple attribute change that provides the ability to increase the scope of the listen queue on waiting or pending TCP connections. For a WebSphere environment with a high transaction rate, this can help greatly because the transfer of information from the listen queue to the TCP input/output buffer space is transferred with little overhead to the TCP stack. With a smaller value, there's more connection processing overhead because of a higher connection establishment time (in other words, new connections can't commence until the listen backlog has sufficient resources).

To view the current setting, use the following no command and attribute query:

 [root@aixserver1: /home]  no -d somaxconn 

The AIX default is 1,024 bytes (or 1KB). This is a small but sufficient setting for smaller environments. As a rule of thumb, for every 500 concurrent users, increase this by 1,024 bytes; with lightweight connections (simply presentation), increase this by 2,048 for every 500 concurrent connections. Lightweight connections rather than Remote Method Invocation (RMI) connections tend to tear up and tear down more often, causing more overhead. Test this setting carefully with your application transaction characteristic.

As a guide, you could safely set this value to 10,240 (ten times the original value) without adversely impacting memory or CPU while still obtaining a good result in increased performance.

To make the changes to this setting, use the following no command and attribute:

 [root@aixserver1: /home]  no -d somaxconn=10240 

TCP Hash Table Size

Similar to the Solaris TCP hash table size, this AIX directive provides some improvement for TCP-based connections, which is the case under WebSphere. The TCP hash table maintains the hash list for TCP Protocol-Control Blocks (PCBs). PCBs, briefly , are the socket-abstracted entities used to maintain things such as TCP socket options, send and receive sequence numbers , windows, segment sizes, and other important information. As such, its ability to handle more information typically means that more information is being buffered or, indirectly, cached.

The default value or size of the TCP hash table in AIX is 24,499. This value is an integer value of the number of hash elements available within the hash table itself.

To confirm the size of the TCP Hash table on your AIX server, use the following command:

 [root@aixserver1: /home]  no -d tcp_hashtab_size 

I recommend increasing this to a value of more than 100,000; however, try to stay at a value less than 150,000. Therefore, to set the value, use the following command:

 [root@aixserver1: /home]  no -d tcp_hashtab_size=120000 

As always, save and test this value with a load-testing tool to ensure that it doesn't interfere with or disrupt other vital system components or processes.

System and Hardware Tuning

Hardware tuning is one of those ambiguous phrases that refer to a wide range of do's and don'ts, specific to the close relationship between the operating system and physical hardware. Given this, this section doesn't cover all aspects of this field of expertise but instead covers specific hardware performance tuning and optimization concepts that benefit WebSphere.

Chapter 5 covered some of these aspects of platform optimization at a theoretical level and showed the platform design options available to three popular WebSphere platform architectures (Sun SPARC, IBM PowerPC/Power4, and x86/x86-64/Itanium). In the following sections, you'll look at several important aspects of system-level tuning and optimization that can significantly improve the overall stability and performance of a WebSphere-based platform.

You'll look at the following items:

• Disk layout approaches

• Memory allocation and optimization

• Network configuration

• File system configuration and selection

Although these aspects may seem somewhat trivial, you'll be surprised at how many WebSphere installations don't always follow them and just how much a difference some of these settings can make.

Disk Layout Approaches

No doubt you've heard , read, or used file system layout structures such as the Oracle Filesystem Architecture (OFA) standard. WebSphere and the underlying operating system can benefit from a similar approach. Although the necessity for WebSphere-based environments to be structured in as methodical manner as the OFA standard isn't needed, there's still a great deal of value to be found in logically laying out your environment.

Note  

This approach isn't endorsed by IBM as a recommended layout; however, it's one I've found to be useful for WebSphere servers.

You'll first look at what data types you have in a WebSphere environment. You can break down the data types as follows:

• Core operating system (for example, kernel and libraries)

• Logging and auditing information

• WebSphere itself (the binaries and libraries)

• WebSphere installed applications

• WebSphere application usage directories

• Third-party applications

• Administration home directories (for example, user directories)

If you look at those seven data types within a WebSphere server, you should notice that the breakdown is split not only on data types but also on usage patterns. For example, once a server has booted , typically the core operating system files aren't frequently accessed (as opposed to WebSphere application files such as JSPs and Java classes).

You'll also notice that there's an obvious split between WebSphere binaries and libraries (in other words, the core WebSphere package) and application data.

The reason for this is fairly simple: The recommended approach attempts to split up components to the point where if one file system filled up because of a core dump or a runaway process dumping lagging information and so forth, the chances for the WebSphere-based applications failing would be minimal ”or, one chance in seven if using this approach.

J2EE and Java applications by nature can quickly dump logs and auditing information to a file system. It's not uncommon with large applications that may have verbose logging activated to dump several hundred megabytes every few minutes.

Other than the sheer amount of disk space to support this rate of file I/O, your WebSphere-based environment needs to be able to identify the vast amount of I/O and then somehow control and manage the rotation of that data.

What happens, however, if your application logs consume too much disk space in too short amount of time, causing a file system to fill up ”one that's shared with applications, WebSphere, and logging information? If WebSphere attempts to passivate EJBs or compile Java code and the file system is full, your application will cease functioning correctly. Further, your customers will start to see an impact on the application's performance.

You need to split log files from application temporary storage, away from file systems that may create things such as file-based semaphores and the like so that you have an almost uninterruptible operating environment or one that's close to it.

Unix File Systems

My recommended approach for a WebSphere environment is to lay out a system using the following structure:

Root and swap on a separate physical spindle (disk) to all other file systems and mirror them.

Although slightly dependant on your Unix variant, the root file system should consist of / , /etc , /kernel , /bin , and /sbin .

Possibly on the same disk, create another mount point for the following data types:

• Application data mounted as /usr

• Another mount point for optional or third-party applications under /opt

• Another mount point for /var

• Another mount point for /tmp

The WebSphere package should sit under something such as /opt , which is separated from the rest of the system data.

The WebSphere logs directory should reside also on its own mount point or file system. Given that this directory can easily become massive with auditing logs and application dumps, it's important for operational support as well as for system continuity to ensure that the logging information is maintained for as long as viably possible. From a performance perspective, if your application is producing a lot of logging data, it may be prudent to log this information to another set of disks. As you know from Chapter 5, you only have a finite amount of I/O available per physical disk. You need to be consistent of this, especially when considering high write-based data requirements such as the activation of debugging and tracing within WebSphere.

Separate onto another file system all user directories. This should be restricted to operational support people (those in your team and who you control or manage). Consider using quotas to help ensure that even people within your team don't go filling up the /home or / export/home directories, causing grief for all other system support people.

If disk space and budgets prevail, consider splitting up the WebSphere installedApps directory. In the previous examples, this would be located in somewhere such as /opt/WebSphere/AppServer/installedApps . This directory houses all your deployed applications. Given that a lot of compilation can occur here, it's wise to try to split this out to another file system for continuity.

I'll summarize these points with a real-world example. Let's suppose you're operating WebSphere on a Linux platform. In this environment, you have several Linux-based WebSphere application servers that operate in a highly horizontally scaled environment.

An example file system layout should provide an extensible and robust layout of file systems. Each file system is extended to a point where it's succinctly separate from the rest of the system, greatly minimizing impact to your operational platform.

Windows-Based File Systems

Windows-based file system layouts are somewhat more simplistic but do follow the same methodology to that of the Unix file systems. The key is to ensure compartmentalization of differing data types. In a Windows world, there are two ways of achieving this breakdown. The first is via the standard C, D, and E drive mapping nomenclature , and the second is the method of using Server Message Block (SMB)-based file systems via the \\machine\mount convention.

The latter method isn't recommended for high I/O requirements such as logging and virtual memory; however, for storing shared application binaries between multiple clustered WebSphere servers, SMB provides a sound solution, both performance-wise and cost effectively.

The basic premise with Windows-based file systems is to ensure that, where possible, each component of the WebSphere environment is compartmentalized.

Although Windows enthusiasts may disagree with me, Windows standard file systems don't yet provide the level of functionality and adaptability as is provided with the standard Unix-like file systems. My recommendation is therefore to extend, while keeping your sanity , the Unix-type layout to be separate drive names . Let's consider that point for a moment.

You should look to ensure that Windows itself, the core operating system, and its virtual memory counterpart are located on a disk that's separate to the application and logging disks.

At worst, you should have four drives ”all mirrored ”and one pair containing the root and swap (virtual memory) services and another pair supporting all the application data such as logs and installed applications directory ( installedApps ).

If you're fortunate to have a RAID array on your Windows-based WebSphere application servers, look to strip these two components as much as you can. Don't be afraid to have a dozen or so drive letters if it's going to help you compartmentalize your operations. Remember, what would you rather have ”a system that falls over because it's out of disk space or a system that has a few more drive letters ?

Network Configuration

As I've discussed earlier in this chapter, memory within WebSphere and J2EE applications is a critical part of the overall performance. You've seen how to tune the configuration of networking within the various operating systems; however, you'll now look at some ways to gain more performance from networking within a WebSphere environment at a physical level.

In Chapter 5, I talked about good network architecture. Like most things, fast, redundant connections are always the best way to go.

If your WebSphere environment is an exceptionally high-loading environment that makes a mockery of a 100Mbps network infrastructure, you may be considering not only upgrading to Gigabit Ethernet but some form of multipathing infrastructure.

All Unix-based operating systems support some form of multipathing technology that allows you to effectively load balance ”or, multiplex multiple network links together ”to gain orders of magnitude increases in your overall networking performance. The beauty of this approach is that you can gain massive amounts of bandwidth for usually only the cost of additional network interface cards and cabling.

For example, Solaris supports a feature called IP Multipathing (IPMP). This essentially uses multiple network interfaces of whatever speed you desire and binds them together to form a single logical interface. This feature is only supported currently on Solaris and Solaris-based servers, but you'll be able to bind two interfaces together with little trouble and gain an 80 “90 percent improvement of the original speed.

For example, binding two 100Mbps interfaces together will provide you with approximately 180Mbps of theoretical bandwidth. That's a fair increase in performance for some extra cabling and another fast Ethernet port! You can tie several of these links together that, theoretically, would allow you to support multigigabit Ethernet networks between hosts .

One thing to note about this technology is that, unlike jumping from 10Mbps to 100Mbps to 1000Mbps where you get a decrease in latency as you step up each level of technology, this doesn't apply the same way as bonding multiple network links together with IPMP.

Both Linux and AIX have similar features that allow you to do the same.

Intelligent Routing

Another feature I've seen work quite well is that of intelligent routing via multiple interfaces. Through the use of somewhat small Time to Live (TTL) values in routing tables, you can set up routing between redundant/multiple switches and redundant/multiple network interfaces so that if a particular network interface appears overloaded, then the operating system will attempt to send the request via redundant paths.

You can also achieve this via a combination of Work Load Management (WLM) with EJBs within WebSphere. I'll discuss this in more detail in Chapter 8 when you'll look at WLM.

If you're looking at operating well-distributed WebSphere components, you can also look at smart routing protocols such as Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF). These protocols, among others, provide sophisticated routing capabilities. These capabilities include intelligent, self-healing routing capabilities, which pretty much do what the previous example suggested by default.

SSL Considerations

Within the WebSphere HTTP Transport configuration, it's possible to configure WebSphere so that it'll communicate with clients or the HTTP plug-in on a Web Server via Secure Sockets Layer (SSL). This may be a necessity for your application environment if communications between various tiers need to be secure.

You need to be cognizant that there's a performance penalty you'll be imposed with on the WebSphere application server. The degree at which this impacts overall application response time and general transaction performance is driven by two factors: first, the type of SSL cipher you want to use and, second, how powerful your application server is (how many CPUs you have, what type they are, and how fast they are).

The following sections discuss a few options that may help you increase your application environment's performance if you're using SSL-based HTTP communications.

Increase Number of Requests per Connections

One of the big hitters for SSL performance difficulties is the initial SSL handshaking for an SSL-based connection. Once the connection in an SSL session has been established, generally speaking, the performance is fine. Depending on your cipher level and type, as well as the specification of your system's CPU(s), you should only notice a small decrease in overall transaction response time with SSL transactions, once established.

On loaded systems or systems where the CPUs aren't as suited for encryption/de-encryption, the bottleneck is in the handshake.

Note  

x86-based CPUs tend to outperform their more expensive Reduced Instruction Set Computer (RISC)-based CPUs such as the Sun SPARC and IBM PowerPC-based processors. There's nothing wrong with having SSL offloaders or Web servers operating on an x86-based platform with the rest of the application environment working from RISC-based CPUs.

Therefore, the goal is to try and push as many requests down an existing connection.

In WebSphere, you're able to set up HTTP keepalives within the HTTP Transport settings. This setting allows you tune the number of requests coming in from the HTTP plug-in on the Web servers so that they (the requests) are contained within the scope of a single connection, up until the point where the maximum requests per connection has been reached.

At that point, WebSphere will drop the connection, and the HTTP server and plug-in will re-establish another one. This helps prevent denial of service attacks; however, it also helps to decrease the setup time for each transaction (the handshaking).

The key with this range of settings is to increase the requests per connection to a reasonable limit. By default, this setting is set to 100 requests per keepalive connection; however, if you're using SSL, look to increase this to 250 and test.

Caution  

Be aware that by changing this setting you're affectively changing the queuing model of your WebSphere environment. I'll discuss WebSphere queuing later in this chapter.

SSL Crypto Interface Adapters

Another option for increasing SSL-based performance is to consider SSL crypto interface cards. These adaptors plug into your server and, via system-level device drivers and plug-in agents to supported applications, allow the SSL encryption and de-encryption to be offloaded to special hardware-based crypto processors. This reduces the impact to the core CPUs and increases overall performance.

Be aware that the crypto card drivers will need to be able to support WebSphere if you're looking to terminate the SSL connections at the WebSphere server or your chosen HTTP(s) server if you're looking to operate them on your HTTP servers.

SSL Ciphers

As you're no doubt aware, there are many SSL ciphers available: DES, RSA, 3DES, MDx, and so on are just a few cipher implementations available.

Different ciphers are processed in different ways, and these different approaches to encryption can affect the performance of your application. Many ciphers also allow for different key lengths to be configured. Unless you're operating a military or high-security WebSphere implementation, there may not be a need for long key lengths.

WebSphere Queues: A Queuing Theory

WebSphere queues are an important part of WebSphere performance and capacity management. WebSphere operates, or can be tuned to operate, with what's known as a queuing network . What this essentially means is that each tier within your application environment is configured to handle a certain amount of requests or transactions or connections at any one time.

WebSphere supports the two key queuing models: open and closed. By default, the closed queuing model is implemented. In this configuration, all the queues within the environment are tightly controlled and more importantly, capped. By setting up your different tiers to have a finite number of connections in a closed queue model, you control and curve runaway processes (such as denial of service attacks) and manage your resources more intelligently.

The open queue model that exists is basically a free-for-all. In this model, the different tiers aren't constrained. My recommendation is not to use an open model. By using the closed queuing model, you can manage your environment with more accuracy and better plan and manage your capacity model.

In Chapter 1, I discussed the carrot and the turnip models whereby your WebSphere environment is meant to accept many connections on the frontend (the Web server) and fewer proportionate connections on the backend. Figure 6-6 shows the carrot model discussed in Chapter 1.

Figure 6-6: Correct queuing configuration, the Carrot model

As mentioned, your environment needs to appear like a carrot in that the thick end (of the carrot) relates to the size of the pipe or the queue at the Web server tier, with the smaller, pointier end relating to the proportional queue size at the data tier.

Between the Web tier and the data tier are several other components, all of which you're now familiar with. Other than the Web and database servers, these components are also essential to making up the overall queue:

• Web server

• Web container

• EJB container

• Object Request Broker (ORB)

• Data sources

• Database

• If implemented, legacy systems integration components (resource connection manager)

You'll now see how a queue operates in a real-world example.

Suppose you had a Web-based application, deployed and operating within WebSphere. For the purpose of this discussion, consider it to be a fairly high-volume site (a high transaction rate and lightweight transaction characteristic) that provides customers with delivery tracking information.

Figure 6-7 later in the chapter shows a queuing network model for the example application. As you can see, the application deployed has a Web tier (thin model web server), a Web container, an EJB container, and a JDBC data source to a backend database.

Figure 6-7: Example queuing model

The size of the bars in the model indicate how many open connections are possible within each tier. For example, the Web container is configured to accept 200 connections at any time.

Why Do You Do This?

Looking at Figure 6-7, you're probably saying, "So what?" The benefits from what this can achieve are really a two-fold return. First, queuing network models such as this allow you to model your environment and understand how many connections are required through your tiers. This translates into dollars per transaction or sizing estimates for your physical architecture.

Second, queuing models help to manage the WebSphere application and servers correctly. You can rate limit the number of inbound connections to any one tier to ensure that all users and customers of the application are experiencing maximum performance.

The method in which queues are meant to function is that the heavier the transaction cost, the less or smaller the amount of concurrent load you want to have as you get deeper into your model. That is, for a large system, you may want to allow for 1,000 connections coming into the Web server, 750 at the Web container, 250 at the EJB container, 100 at the JDBC connection manager, and 75 at the database server.

The key is to try and reduce or minimize the number of requests within the WebSphere application server at any one time. The aim is to try and keep the pending requests in the network, usually out front of the Web server. This approach works best when you have multiple channels or WebSphere application servers.

By keeping requests pegged at a certain measure, it allows for a more stable environment where customer or user requests aren't bogging down a system. That is, you have a fixed number of requests in each component or tier at any one time. This is controllable and manageable.

Overall, this progressive queuing approach promotes solid system design and management as well as ensures response times are optimal.

That all said, you may actually find that there's a need in your environment to have a one-to-one relationship between Web container connections, EJB container connections, and database/data source connections. This is still using a queuing model and is valid, but it isn't a "lean" model. If you have large resources, this approach is achievable, but you should still ensure that you model and baseline the queue settings ”whatever they may be ”so that there's a control and capacity modeling measurement.

You'll now look at how to calculate the queue sizes.

Determining Queue Sizes

Determining the size of each of the queues is a several-stage process. It can be summarized as follows:

1. Understand and identify your user requirements (for example, concurrent users and so on).

2. Set an initial estimated queue overlay on the overall system.

3. Test and measure the initial overlaid figures to obtain a baseline.

4. Analyze the results of baselining and the undertaking of a tuning effort to configure the queues to the optimal size.

The first question you need to answer, and the one you should know already, is how many concurrent users are you expected to have online at any one time?

Let's assume that 500 users is the magic figure, with each session lasting ten minutes on average, and five functional transactions are completed throughout that ten minutes.

You could make an initial assessment and say that at any one time (or within the space of a minute), there would be approximately 250 concurrent functional transactions taking place (for example, 2,500 transactions over a period of ten minutes).

You'd know that each transaction would require resources from the Web server and Web container and one in every two transactions would require database query or updates.

The model would look like that in Figure 6-7.

This initial assessment for a WebSphere environment is sufficient from a starting point, but you haven't taken into account the actual latency between each tier.

Caution  

Figure 6-7 is an example only, and the settings shouldn't be transposed or used directly within your environment. Be sure that you follow the guidelines within this chapter to ensure that your queuing model is correctly configured.

For example, what if a basic transaction that was made to the database (such as a query of some sorts) took 5,000 milliseconds (five seconds) to respond? This would limit the transaction throughput for your overall application and cause requests to "bank up" in front of the EJB container.

To a point this may be acceptable; however, given that each tier in front of the EJB container would need to manage how it handles the queued connections, you'll need to understand what will happen with those queued connections.

If you had a load balancer in front of the Web servers, the request could be queued in the network, further back in a Web proxy, or even at the user's browser. This queuing helps to ensure that each piece of your application environment is operating at a level that's adequate to meet Service License Agreements (SLAs), and no one piece is or can be overloaded (in theory).

Looking back at the initial baseline queue configurations, it's important that the right tools are used for the task of profiling and baseline identification.

I'll cover the topic of profiling and tooling in Chapters 14 and 15. For usage and detailed information about these tools, how to use them, and the methodology for profiling, review those chapters. For the purposes of this chapter, I'll discuss what items need to be monitored when profiling.

Queue Configuration Baselining

After you've configured the various components within your environment with the chosen baseline queue settings, it's time to profile your application to see how they ”the settings ”respond. For the most part, this type of monitoring and testing works best using the WebSphere Resource Analyzer (WebSphere 4) and Tivoli Performance (WebSphere 5) tools.

In both tools, what you're looking at is the specific element within the environments that corresponds to the queues. For each application server, Table 6-5 highlights the components that need to be monitored to obtain the data.

Table 6-5: Queue Modeling: Items to Monitor

Queue Component	WebSphere Element to Monitor	Level of Monitoring Setting
Web container	Servlet Transport : available threads, active threads	High
EJB container	EJB container: active threads	High
Data source provider	Chosen data source provider: connection pool used, connection pool free	High

Note  

Because these measure only the WebSphere application server, you'll need some tool or tools to monitor the Web server and database server for their concurrent connection rates and resource exhaustion rates. I'll discuss the tools for Web and database servers in Chapters 13 and 14.

You'll now look at the configuration items for each of the main components and what you should look for in terms of correct tuning and optimization techniques.

Updating and Configuring the Queues

For the WebSphere-based components, you can make the changes to the various queues via the administration console. As I've mentioned throughout the book, these changes need to be made in a logical and structured manner. Make a single change at a time, test, and change the next setting ”if required. Never change several settings a time. The reason for this is so that you know what each setting change has done and how it affects your system. In this fashion, you can measure the effect, which allows you to better understand the characteristics of the environment.

By also changing one setting at a time, you avoid the chance that one of your tuning changes results in negative performance. If you change several settings at once, you won't have any idea which setting change affected the performance negatively.

I'll discuss making changes to the queues and modeling their performance in Chapter 14. You'll now look at some example settings for the primary tiers in a queuing model.

HTTP Web Server

The Web server queue settings amount to the number of connections that the Web server will accept or process concurrently. All the HTTP servers I've discussed in this book allow you to configure the maximum number of concurrently processed requests.

This setting should be larger than the maximum number of concurrent sessions (for example, logged-in users) to your WebSphere-based applications. For example, if your capacity model states that during peak periods, your environment will have to support 500 concurrent users, this setting will need to be approximately 10 percent more than 500 (for example, 550 sessions).

The additional 10 percent will cater for unforeseen peak loads or issues where those impatient users click Stop on their browser and quickly rerequest the page. You may also have an application that requires (or drives) users to open multiple concurrent sessions. This will also increase your concurrent session usage.

WebSphere Web Container

The next major tier component within the queue is the Web container. The key to this queue setting is to the minimum and maximum HTTP thread pool values within the Web container settings.

As you explored in earlier chapters, this setting should reflect the number of concurrent connections active from the HTTP Web server (via the plug-in) to the Web container. This needs to reflect the number of concurrently accessed servlets, JSPs, and other content components such as HTML, images, and so forth.

Therefore, you should determine this setting using the following:

 Maximum Threads = Average_Number_of_Components_per_Client_Transaction    Number_of_Concurrent_Sessions 

Using the Resource Analyzer or Tivoli Performance tool (WebSphere 4 and 5, respectively), you can monitor this queue setting via the Active Threads counter. This will provide you with a real-time view of how many threads are being used within this queue tier.

If this counter is constantly hovering or constantly at the maximum thread pool size value (as configured in the Web container services settings), this value may be too low. Ideally, you want this counter to sit at approximately 80 percent of the total pool size.

WebSphere EJB Container

The EJB container queue tier settings aren't different from that of the Web container thread settings. The EJB container's primary queue setting is derived from the ORB thread pool. This setting, as you've seen, is the number of concurrently active requests being processed by the EJB container. Not unlike other thread settings, this value is derived by understanding the average number of requests per client transaction that will be accessing the EJB container.

For example, if for every two Web container-based requests, a resulting EJB container request for an EJB object took place, with a Web container configured with 200 queues, the EJB container would be configured, initially, with 100 available queues. Of course, if your environment isn't running a Web-based UI and is running a thick-Java based UI, this queue value is predominantly the primary " throttle control" into your WebSphere environment.

The equation to set this queue setting within a thick-Java UI environment isn't complicated ”essentially you're calculating the average number of concurrent EJB requests against the concurrent users in your environment. Be aware that, assuming you're only accessing your data tier from the component within the EJB container, setting the queue value for this tier too high may affect the performance (in other words, overload) the database.

Use the queue settings of the JDBC connection pool manager to help rate limit this problem.

Using the Resource Analyzer or Tivoli Performance tool (WebSphere 4 and 5, respectively), you can monitor this queue setting via the ORB Thread Pool counter. Like the Web container, as you tune and test these settings, use this counter to understand how this queue setting is working against the maximum thread pool size.

Again, like the Web container, my recommendation is to set the queue setting to be 80 percent of the modeled maximum number of concurrent threads.

JDBC Connection Pool Manager

The data source or JDBC pool manager queue is the next queuing tier in the overall model. By now, you're probably seeing a similarity between all the different queuing tiers in that the way to tune and monitor them all is via the respective pool sizes. In the case of the JDBC pool manager, or data source manager, the key setting to test and tune is the Maximum Pool Size setting within the pool manager configuration sections of the WebSphere console.

Prior references to the JDBC pool manager settings within this book have suggested that this value of the maximum connection pool size should be derived by adding 5 to 10 percent on top of the value of the ORB thread pool value. Other references suggest that this value should be simply the ORB thread pool size plus one. I don't agree with this rule because it limits or reduces the ability for the container to properly handle peak loads efficiently.

When you're tuning and modeling this value, within the Resource Analyzer or Tivoli Performance Viewer tool (WebSphere 4 and 5, respectively), you can monitor the queue setting via the Average Pool Size counter.

You should monitor this value to remain under the maximum pool size value for 80 “90 percent of the period in which your testing is performed.

Tuning the JVM

The JVM is one of the most important parts of your overall WebSphere environment from a performance perspective. You can tune WebSphere to an inch within perfection , and yet still, a poorly configured or tuned JVM will bring a system to its knees.

Through the optimization efforts discussed so far and those coming in later chapters, you can somewhat insulate a badly performing JVM. However, over time, it'll catch up with you.

The JVM is, as discussed, a virtual machine environment, specific to each platform. The JVM is responsible for all matters dealing with things such as memory allocation and deallocation (but don't let this make the developers become complacent!), object creation and life cycle management, and runtime control of the Java byte code.

Given that Java is an object-relational language, it's structured around the premise that live objects exist and are managed in memory. Although object-orientated environments provide powerful architectural paradigms , they also exhibit a fair degree of memory-intensive characteristics.

I've discussed in earlier sections and chapters that for the JVM to support a managed memory allocation/deallocation model, it must ensure that regular housekeeping is conducted . The JVM is therefore a sophisticated runtime engine, providing a great deal of autoplumbing and execution capabilities that would normally be managed by the developer in other object-oriented languages.

One of the side effects of having a self-managing environment such as this is that the JVM, as configured, doesn't provide a one-size-fits-all model. Each different application will require, or should require, different memory allocations and configuration setups. What's known as the Java heap is the memory space that's allocated for runtime for your WebSphere-based applications. This heap space is where runtime code is managed as well as where data and objects are stored. Your application type and ultimately its load will determine what sort of configuration your JVM will require.

To further complicate matters, there are many vendor-specific JVMs on the market. Generally speaking, the primary setting or available configurations are standard; however, there are a number of unique capabilities and options that each individual JVM supports.

JVM Tuning Options

You'll now look at the primarily used JVMs on the market. Each of the three featured JVMs support the core JVM functionality, but each also supports its own specific tuning and performance options.

I'll cover the following JVMs:

• Sun HotSpot JVM

• IBM JVM

• BEA JRockit JVM

Sun HotSpot JVM

The Sun HotSpot JVM is the JVM offering from Sun Microsystems. It offers a generational garbage collection mechanism that considers object lifetime. Essentially, the heap is split into the New Heap (also known as Young Heap ) and the Old Heap.

The New Heap is then split into three subsections: the first known as Eden and the other two are Survivor heap spaces. The way that this JVM object life cycle works is that the new objects are created in the Eden space. As the objects move through their usage life cycle, they're moved out of the Eden space into the Survivor spaces and, ultimately, into the Old Heap.

It's important to note that when the New Heap fills up, a minor garbage collection takes place in the New Heap space. In this minor collection, all surviving heap-based objects are moved into the Old Heap. Once the Old Heap fills up, a major collection takes place ”this is analogous to a normal garbage collection.

The JVM bootstrap parameters that affect these settings are as follows:

• -XX:SurvivorRatio: This value determines the Eden to Survivor ratio.

• -XX:NewRatio: This value determines the Old to New Heap ratio.

• -XX:NewSize: This value determines the new initial bound.

• -XX:NewSizeMax: This value determines the new maximum bound.

You need to carefully consider these values if you're determined to change them. My recommendation is that for 80 “90 percent of all cases, managing the minimum and maximum heap sizes for this JVM will provide sufficient tuning capability for your JVM sizes.

What these values do provide for is the ability to fine-tune garbage collection for application environments that are object "savvy." That is, if your application shows characteristics of heavy object creation and usage, these settings may help by fine-tuning the frequency of garbage collection.

IBM JVM

The IBM JVM offering supports a number of parameters that can improve performance of object allocation within the JVM heap. There are several available options to set; the following list provides an overview and several recommendations of when and where to set these optional parameters:

• -Xgcthreads: This value allows you to set the number of threads for garbage collection. It allows you to get around the standard single-threaded model that pre-Java 1.4 JVM's have. As a guide, this value can be set to the number of CPUs that your system has. For example, if your system is a quad-processor system and you expect your JVM to have moderate load, configure this value to be 2 to 4 depending on your collection frequency and demand.

• -Xml: This value defines the maximum object size for limiting the size of objects to be allocated and referenced from the local cache. The key is to use the local object cache as much as possible rather than the general JVM heap for small- to medium-sized objects.

These values, when tuned appropriately, tend to drive performance improvement by up to 20 percent for fairly standard Java environments.

As discussed many times in this book, be sure that you size your object allocation and memory usage carefully to ensure that settings such as these are optimized appropriately for your specific WebSphere application environment needs.

The BEA JRockit JVM is a Wintel platform (x86 Intel Itanium) optimized JVM. Originally developed by a European company, JRockit was recently released by BEA (the makers of the BEA WebLogic J2EE application server). Some developers and system architects say it's the "world's fastest JVM."

JRockit is an unusual JVM implementation and is quite different from others; it offers an advanced garbage collection architecture via four different collection methodologies. Two of the key garbage collection facilities within the JRockit JVM allow for simultaneous collection and object cleanup in parallel to the main Java JVM threads. This can dramatically increase the overall runtime performance by not pausing the JVM runtime for the garbage collection thread. Of course, this capability only really makes a real impact with multiple CPUs, but the improvement in performance is evident still via a single CPU system.

The second major difference between JRockit and other JVMs is that JRockit also provides an advanced Just In Time (JIT) compiler. The feature essentially precompiles all methods within all classes within your application when boot-strapping, and it provides a continual background optimization function that constantly removes bottlenecks. Lastly, the JVM's optimization thread capability constantly tunes frequently used methods and interfaces within deployed Java code.

In summary, if you're running an Intel-based or AMD-based platform (32-bit or 64-bit), then consider the JRockit JVM if your application is memory intensive (heap usage and object allocation).