Tuning the Application Server

WebLogic provides many server-specific performance-related configuration settings. For instance, enabling native I/O on specific platforms greatly improves the efficiency with which WebLogic can carry out I/O operations. WebLogic provides various execute queues that serve as a pool of ready threads operating on a first-come-first-served fashion. You can adjust the Thread Count attribute of an execute queue to set the number of execute threads that can operate concurrently. An execute queue then can be shared by multiple applications deployed to WebLogic. Alternatively, you can create custom execute queues and assign them to specific applications. In this way, you can guarantee that critical applications always can rely on a fixed number of threads.

WebLogic lets you specify when an execute queue is considered to be overwhelmed. This threshold is specified in terms of a user-defined percentage of the maximum length of the execute queue. The server detects an overflow condition when the execute queue exceeds this threshold value, and changes its health status to "warning." You can configure the server to allocate additional threads to complete the outstanding work in the queue under these conditions.

WebLogic also can detect when threads in the execute queue are "stuck" i.e., when a thread is unable to complete its current task or accept new tasks. Depending on the execute queue, the server's health state is changed to either "warning" or "critical." You can adjust the thread detection behavior by varying the time interval after which WebLogic periodically scans the queue for stuck threads. In addition, you can specify the allowed backlog of connections for any server. This restricts the number of TCP connections that WebLogic accepts into its wait queue, before refusing additional requests.

The combination of these settings plays an important role in enhancing the server's performance.

15.2.1 Execute Queues

In WebLogic, an execute queue represents a named collection of worker threads for a server. Dedicated execute queues can be made available to servlets, JSPs, EJBs, and RMI objects deployed to the server. Any task that is assigned to a WebLogic instance is first placed into an execute queue and then assigned to an available server thread on a first-come-first-served basis.

The default execute queue, weblogic.kernel.default, is preconfigured for each WebLogic instance. If a server is in development mode, this queue defaults to 15 threads. If the server is in production mode, it defaults to 25 threads.[2] Unless you create additional execute queues and assign them to specific applications, all web applications and RMI objects rely on this default execute queue.

[2] In WebLogic 7.0, the default size of the execute queue in production mode is 15.

WebLogic also depends on two queues reserved for administration purposes only. The weblogic.admin.HTTP execute queue belongs to the Administration Server and is dedicated to the Administration Console. The weblogic.admin.RMI execute queue is available on the Administration Server and all Managed Servers. It too is reserved for administration traffic only. Neither of these execute queues can be reconfigured in any way.

15.2.1.1 Creating an execute queue

You can use the Administration Console to configure a new execute queue for a server. Choose the desired server instance from the left pane. Right-click the server's name and choose View Execute Queues to view the execute queues for the server. The resulting page also provides a "Configure a new Execute Queue" option. Table 15-2 provides a description of the configuration settings available on this tab.

Table 15-2. Configuration settings for an execute queue

Setting	Description	Default
Queue Length	This setting defines the maximum number of simultaneous requests that can be held in the queue. WebLogic automatically doubles the queue size when the maximum queue length is reached.	65536
Queue Length Threshold Percent	This setting determines the threshold beyond which the server indicates an overflow condition for the queue. It is specified in terms of a percentage of the queue length (1-99). When the actual queue length exceeds this threshold value, this indicates an overflow condition. WebLogic logs an error message and provides additional threads to reduce the workload on the execute queue.	90
Thread Count	This setting defines the number of threads assigned to this queue.	15
Threads Increase	This setting determines the number of worker threads WebLogic may create to resolve an overflow condition on the execute queue. By default, WebLogic merely changes the health status of the server to "warning" without providing any additional threads to handle the overflow. Note that once WebLogic creates the additional threads to handle the queue overflow, the new threads remain in the execute queue until the server reboots. In general, you should examine application setup to determine the reasons for the overflow and modify the thread count to avoid the overflow condition in the future.	0
Threads Minimum	This setting determines the minimum number of threads WebLogic should maintain in its execute queue to prevent unnecessary overflow conditions.	5
Threads Maximum	This setting determines the maximum number of threads in the execute queue. It prevents WebLogic from continually adding more threads to the queue in response to overflow conditions.	400
Thread Priority	This setting specifies the priority for threads in the execute queue.	5

Once you've applied your changes to the settings for the new execute queue, you need to reboot the associated server in order for the changes to take effect.

15.2.1.2 Assigning execute queues to J2EE applications

By default, all applications deployed to WebLogic (except JMS producers and consumers) use the server's default execute queue. After creating a new execute queue on a server, you still need to ensure that the applications you run on the server can use this execute queue. Fortunately, WebLogic lets you assign an execute queue to servlets, JSPs, EJBs, and RMI objects. In order to associate the execute queue with a servlet (or JSP), you need to specify the wl-dispatch-policy initialization parameter for the servlet (or JSP) in the web.xml descriptor file. The following code sample shows how to assign the execute queue mySpecialQueue to a particular JSP page:

 MyServlet
 /critical.jsp
 
 wl-dispatch-policy
 mySpecialQueue
 

In order to assign an execute queue to an RMI object, you must specify the -dispatchPolicy option when using WebLogic's RMI compiler (rmic). Here's how you would assign the execute queue mySpecialQueue to an RMI object:

java weblogic.rmic -dispatchPolicy mySpecialQueue ...

In the same way, use the -dispatchPolicy option when invoking WebLogic's EJB compiler to assign the execute queue to an EJB. WebLogic's EJB compiler implicitly passes the -dispatchPolicy argument to the underlying RMI compiler. In WebLogic 8.1, use the dispatch-policy element in the EJB's weblogic-ejb-jar.xml descriptor to set the execute queue:

 MyEJB
 ...
 myEJBQueue

Custom execute queues are supported for all EJB types session beans, entity beans, and MDBs.

The dispatch policy for an MDB may be ignored if the MDB is attached to a foreign JMS destination.

At runtime, WebLogic allocates worker threads for your servlets, JSPs, EJBs, and RMI objects from their configured execute queues, thereby guaranteeing that selected objects in your application have access to a fixed number of server threads. For those objects for which no execute queue is assigned, the threads will be allocated from the server's default execute queue.

15.2.1.3 Monitoring active queues

To monitor the active queues using the Administration Console, right-click a server and select View Execute Queues, then navigate to the Monitoring tab. Alternatively, select the Monitoring/General tab and then choose the "Monitor all Active Queues" link. This will provide an indication of each execution queue, including the queue length, the number of idle threads, and the throughput. Clicking any queue will let you monitor the threads on that queue. You also can determine the queue length graphically, along with throughput and memory usage indicators, by selecting the Monitoring/Performance tab for any server.

Monitoring the queue length is particularly important. For instance, if your queue length increases over time while your throughput stays the same, any new incoming requests will have to wait longer and longer for the next available thread. Some kind of tuning undoubtedly will be needed in these circumstances. If this situation persists even after tuning and reconfiguration, your server setup may have simply reached the limit on the amount of traffic that it can handle.

15.2.2 Server Threads

The runtime behavior of an execute queue is influenced by a number of parameters, and optimal server performance depends on the optimal configuration of these parameters:

• The thread count for an execute queue
• The percentage of execute threads that will act as socket readers
• How execute queues react to overflow conditions
• How often WebLogic detects "stuck" threads in the queue
• The size of the wait queue, or the number of TCP connections that are accepted before WebLogic refuses additional requests

Let's look more closely at the implications of these parameters on the performance of WebLogic's server threads.

15.2.2.1 Configuring the thread count

The thread count for an execute queue determines the number of allocated threads that can be used concurrently to serve application tasks that utilize the execute queue. You can use the Administration Console to modify the thread count for the default execute queue on any WebLogic instance. Choose the desired server from the left pane, right-click it, and select the View Execute Queues option to display all available execute queues for the chosen server. From the Name column, click the name of the default execute queue, weblogic.kernel.default, to view or modify its configuration settings. You now can raise or lower the default Thread Count value for the execute queue. Once you've applied your changes to the execute queue, you need to reboot the particular server for those changes to take effect.

You must be careful when allocating more server threads to the default execute queue. Threads are valuable resources and they consume memory. You may degrade server performance by increasing the thread count to an unnecessarily high value. Too many execute threads means more memory consumption, and thus more context switching among the available threads. The ideal value for the number of execute threads will depend on the nature of your application. In general, you need more execute threads if the client relies on RMI method calls for a significant amount of its work, or if the application relies on a number of database calls for processing client requests. On the other hand, if your application makes short calls with rapid turnover, a small number of execute threads may also improve server performance.

The ideal value for the queue's thread count can best be determined through trial and error more accurately through observation under test conditions in which all applications in the execute queue are operating under high load. You need to continually raise the number of threads in the execute queue and repeat the same load test, until you've attained peak throughput for the queue. At some stage during the repeated executions of the load test, the thread count for the execute queue will exceed a threshold, beyond which the memory consumption of the threads and excessive context switching actually degrade server performance.

15.2.2.2 Threads as socket readers

Ideally, a WebLogic instance should rely on the native I/O implementation provided for the server's OS. If you must rely on WebLogic's pure-Java socket reader implementation, you still can improve the performance of socket communication by configuring the appropriate number of execute threads that will act as socket readers for each server (and client). You can use the ThreadPoolPercentSocketReaders attribute to specify the maximum percentage of execute threads that are used to read incoming requests to a socket. By default, WebLogic allocates 33% of server threads to act as socket readers its optimal value is again application-specific.

By ensuring that you've configured an adequate number of threads to act as socket readers, you enhance the server's ability to accept client requests. However, finding the right balance between server threads that are devoted to reading requests received on a socket and those threads that actually perform the required tasks is a matter of trial and error. Again, you need to use the Administration Console to set the maximum percentage of server threads that will act as socket readers. Choose the particular server from the left pane, and then select the Configuration/Tuning tab from the right pane. You now can specify a number (between 1 and 99) for the Socket Readers attribute. The number of Java socket readers is defined by the percentage of the total number of execute threads. The Execute Threads attribute represents the total number of execute threads.

You even can configure the number of socket reader threads for the JVM on which the client runs. Use the -Dweblogic.ThreadPoolSize option to specify the size of the client's thread pool i.e., the number of execute threads available to the client. The number of socket readers then can be expressed as a percentage of the size of the thread pool.

The following example illustrates how to specify the number of socket reader threads for the client's JVM, through the -Dweblogic.ThreadPoolPercentSocketReaders option from the command line:

 java -Dweblogic.ThreadPoolSize=6 
 -Dweblogic.ThreadPoolPercentSocketReaders=33 ...

15.2.2.3 Queue overflow conditions

We have seen how WebLogic can detect and resolve potential overflow conditions in an execute queue. An overflow occurs when the current size of the execute queue exceeds a certain threshold. This threshold is specified in terms of a certain percentage of the maximum length of the execute queue. When this threshold value is exceeded, WebLogic changes the server's health status to "warning" and optionally allocates additional worker threads to reduce the workload on the execute queue. These additional threads are introduced to bring the execute queue back to its normal operating size.

Thus, in order to configure overflow detection and resolution, you need to configure the following parameters for an execute queue:

• The threshold percentage that indicates an overflow for the server's execute queue.
• The number of additional server threads that must be allocated to resolve the queue's overflow condition.
• The minimum number of threads that always should be available to the execute queue, and the maximum number of threads that may be assigned to the queue at any time. The upper limit ensures that WebLogic doesn't continually assign new server threads in order to handle repeated occurrences of queue overflow.

Table 15-2 lists the configuration settings that determine how queue overflow conditions ought to be handled. Remember, additional server threads assigned to the execute queue for the purposes of resolving the overflow remain in the execute queue until the server shuts down.

15.2.2.4 Detecting stuck threads

WebLogic automatically detects when a thread assigned to an execute queue becomes "stuck." A thread becomes stuck when it cannot complete its current task or accept new tasks. The server logs a message every time it detects a stuck thread. When all threads in the execute queue become stuck, WebLogic changes the server's health status to "warning" or "critical," depending on the execute queue. If all threads allocated to the server's default execute queue become stuck, WebLogic changes the server health status to "critical." For all other execute queues, WebLogic changes the health status to "warning." This includes any user-defined queues and the weblogic.admin.HTTP and weblogic.admin.RMI execute queues. Recall that the Node Manager can be configured to automatically restart a Managed Server whose health status is "critical." In addition, you can write log listeners that can alert you if this condition arises.

At regular intervals, WebLogic inspects the server threads assigned to the execute queue to determine whether any of these threads are stuck. You can adjust the frequency with which WebLogic checks for stuck threads, and the elapsed time after which WebLogic marks a thread as stuck.

WebLogic lets you configure thread detection behavior only on a per-server basis rather than on a per-execute-queue basis.

Select the desired server from the left pane of the Administration Console and then choose the Configuration/Tuning tab from the right pane. Here you can adjust the following configuration settings:

Stuck Thread Max Time

This setting determines the duration (in seconds) for which a server thread must continually be busy before WebLogic marks the execute thread as stuck. By default, WebLogic marks a thread as stuck if it has been busy for 600 seconds.

Stuck Thread Timer Interval

This setting determines the frequency with which WebLogic periodically scans execute queues for stuck threads. By default, WebLogic scans for stuck threads every 600 seconds.

Once you've applied your changes to these settings, you need to reboot the particular server for these changes to take effect.

15.2.2.5 Monitoring active server threads

To monitor the active server threads using the Administration Console, right-click a server and select View Execute Threads. This will provide an indication of each server thread, the execute queue that is associated with the thread, and further information such as the total number of requests that have been handled by the thread. Alternatively, you can select the Monitoring/General tab and then the "Monitor all Active Queues" option. This enables you to view server threads on a per-queue basis.

15.2.3 Socket Connections

WebLogic supports IP-based socket communication in many scenarios. For example, all remote RMI communication uses sockets, as does internal cluster communication such as state replication.

WebLogic supports two socket configurations: a pure-Java implementation and a native socket reader implementation. If you can, always use the native implementation because it improves the performance of socket-based communication considerably. The pure-Java implementation uses threads that must actively poll all open sockets to determine whether they contain data that needs to be read. The same problem is faced by developers who build applications using sockets on the JDK 1.3 platform. The threads reading from the sockets are always polling the sockets, even if there is no data to be read. This severely reduces the performance of these applications.

JDK 1.4 introduces a mechanism whereby a socket reader can be asynchronously notified when there is data, and hence the socket readers do not have to waste CPU cycles polling. A similar technique is used in WebLogic's native socket implementation. This makes the native socket reader implementation much faster than the pure-Java implementation. As applets don't have access to the native code implementation, they cannot use it, so the efficiency with which they can perform socket I/O is still limited. BEA's web site lists all of the supported platforms for WebLogic, together with availability information regarding the native service pack.

In order to enable the native socket implementation for a server, choose the server from the left pane of the Administration Console and move to the Configuration/Tuning tab. The Enable Native IO option already should be selected; this option is enabled by default.

15.2.3.1 Optimizing the pure-Java socket implementation

The pure-Java socket implementation can be optimized somewhat by ensuring that there are always enough reader threads available on a WebLogic instance. Too few reader threads may degrade performance because of unnecessary waits while the reader threads eventually get around to polling a socket that does have data. Of course, too many reader threads will unnecessarily consume more resources.

The number of reader threads necessary for a WebLogic instance is highly contextual. If the server belongs to a combined-tier cluster, each server potentially will open a maximum of two sockets, which it needs to replicate any session state. Both web applications and EJB components will be deployed homogeneously to the cluster. The collocation optimizations mean that method calls from servlets and JSPs to EJBs always will be directed to local replicas of the EJB objects. Therefore, no additional socket reader threads will be needed.

If a cluster has a pinned object, you will need an additional socket reader on each member of the cluster, which can be used to reach that pinned object. If you've designed a multi-tier application setup, such as that depicted in Figure 14-1, each member of the web/presentation tier will potentially communicate with each member of the object tier. A clustered object may be fetched from any member of the object tier, depending on how the load balancing occurs. In addition, a server in the web tier will need to communicate with other members for state replication. This means that each server in Figure 14-1 needs three sockets for this communication. If a cluster also will be accessed by external Java clients, you need to take these into consideration as well.

To adjust the number of socket readers on platforms on which there are no available performance packs (or native socket reader implementations), choose the server from the left pane of the Administration Console and modify the Socket Reader setting from the Configuration/Tuning tab. This number determines the percentage of the total execute threads that will be dedicated to socket readers.

15.2.3.2 Client socket readers

External Java clients that communicate with a server or cluster generally will use the pure-Java implementation of sockets. You can configure the number of socket readers on a client using two command-line options that specify the thread pool size and the percentage of the thread pool that may be used for the socket readers. The following example shows how to specify the number of socket readers at the client's end:

java -Dweblogic.ThreadPoolSize=20 
 -Dweblogic.ThreadPoolPercentSocketReaders=40 ...

15.2.3.3 Managing TCP connections

For each WebLogic instance, you can use the Accept Backlog parameter to set the number of TCP connections WebLogic accepts before refusing additional connections. This parameter determines the maximum number of TCP connections that may be buffered in the wait queue. By default, WebLogic accepts 50 TCP connections before refusing additional connection requests. The Accept Backlog configuration setting can be located from the Administration Console by navigating to the Connections/Tuning tab for the selected server. The maximum value for this setting is OS-dependent.

WebLogic must queue up client requests for socket connections when the deployed applications aren't accepting those connections fast enough. For example, this might occur during excessive server load or during lengthy garbage collection cycles. If clients keep getting "connection refused" messages when trying to access the server, it may be because your accept backlog is not large enough. In this case, keep increasing the value until your clients no longer get the messages.

15.2.4 Tuning a Clustered Setup

As we saw earlier in Chapter 14, a WebLogic cluster represents a group of servers working together to provide scalability and high availability for all kinds of applications and services deployed to the cluster.

WebLogic offers scalability to J2EE applications in a way that is transparent to both end users and developers. Scalability refers to the system's capacity to grow in specified dimensions, as more demands are placed on the system. Typical dimensions include the number of concurrent users at any time, the number of transactions that can be handled in a unit of time, the average response times, and many more. In case of a well-designed application, you can improve performance by simply adding another server to the cluster. WebLogic automatically balances the load between the available members of the cluster. Thus, a WebLogic cluster allows you to add more capacity to the middle tiers, without affecting any existing clients of the application.

In addition, the cluster provides high availability by including multiple redundant servers that host duplicate copies of applications and services. This way, a client is insulated from individual server failures. In case of stateful or pinned services, WebLogic provides replication services that enable a peer server to take control of the service in case the primary server fails. This too enhances the availability of applications and services to clients.

A cluster configuration that provides (near) linear scalability for your J2EE applications is ideal. This means you should tune your setup to ensure that the system can grow along specified dimensions proportionately as more servers are added to the cluster. In general, you should isolate any application-specific performance issues before optimizing your application setup in a clustered environment.

Chapter 14 provides an in-depth discussion of the performance implications of adopting combined-tier architectures in favor of the multi-tier application setup. It presents several scenarios in which you need to balance the flexibility of your application setup against the performance of your applications. It also covers the impact of IP multicast and socket-based communication on the overall traffic flowing through your network infrastructure.

15.2.4.1 Multi-CPU machines

A multi-CPU machine may host a cluster of WebLogic instances, or perhaps even multiple clusters. In this scenario, the ratio of the number of WebLogic instances to the number of CPUs becomes a critical factor. In order to determine the ideal ratio for a CPU-bound application, you should first set up performance tests for your application using a ratio of 1:1 that is, one WebLogic instance for each CPU. If you discover that CPU utilization is at (or very near) 100%, repeat the same tests using a higher ratio of number of CPUs to servers say, one WebLogic instance for every two CPUs. Remember that in a production environment, a server is sporadically involved in administration tasks as well. Thus, you should set aside a limited CPU capacity for handling the administration tasks under load. BEA holds the view that 1:2 is the optimal server-to-CPU ratio i.e., one WebLogic instance for every two CPUs, ignoring the individual processing needs of a particular application. Your mileage may vary, though.

Of course, before you determine the optimal ratio of the number of servers to the number of CPUs, ensure that your application is truly CPU-bound. For network I/O-bound applications, you may in fact improve performance by replacing the existing NIC with a faster one, rather than bolting on additional CPUs. For disk I/O-bound applications, you may improve application performance by upgrading the number of disks or controllers, or simply by installing faster disks. Thus, you should consider the ratio of the number of servers to CPUs only after verifying that your application is indeed CPU-bound.