Tuning the Core Server Performance: The Thread Pool

WebLogic Server is a Java-based multithreaded application server. The processing tasks of WebLogic Server are conducted in parallel by multiple threads known as execute threads inside WebLogic Server's thread pool . By default, WebLogic Server allocates 15 execute threads to the thread pool, which, for most application scenarios, is a sufficient thread count.

A percentage of these execute threads (the default is 33%) is reserved for socket-reading purposes, and they are appropriately known as socket reader threads . These socket reader threads read inbound work requests from sockets and dispatch them to an appropriate execute queue . The remaining percentage of the execute threads performs the actual processing work inside WebLogic Server; hence, these threads are termed worker threads . Worker threads pick up work requests from the execute queue and process them.

As a WebLogic Server administrator, you are responsible for finding the optimum balance between the number of execute threads you configure as socket reader threads and those you configure as worker threads inside the WebLogic Server thread pool.

Note

The terms execute threads and worker threads can be used interchangeably to denote threads in WebLogic Server where work requests are executed.

The Socket Reader Implementation in WebLogic Server

WebLogic Server provides two types of socket reader implementations :

• Java socket readers This is a pure-Java implementation of the socket readers and hence portable. In this implementation, the socket reader threads must actively poll all the opened sockets on the server to determine whether there is any data to read off them. Polling becomes expensive because time is wasted in polling inactive sockets that have no data. The problem is magnified when there are more open sockets than socket reader threads. In this scenario, each Java socket reader thread must poll more than one open socket, which lets an active socket go unserviced until a socket reader thread polls it.

• Native socket readers This is an efficient native socket reader implementation specific to your operating system. In this implementation, the socket reader threads are notified (via an interrupt) when a socket becomes activefor example, when there is data to read at the socket. The efficiency comes from the fact that the socket reader threads do not poll all the sockets (active and inactive) and service only the active sockets.

For information on the use of socket readers in a WebLogic cluster, see "Understanding How WebLogic Servers Communicate within a Cluster," p. 845 .

For optimum socket reader performance, you should use the native socket readers. WebLogic Server comes bundled with a platform-specific performance pack, which allows the native socket readers to make use of platform-specific asynchronous system calls to read data. The Java-based socket reader implementation is synchronous and can be slow.

Note

Performance packs use a platform-optimized, native socket multiplexor to improve server performance.

You can use native socket readers with all the primary platforms supported by WebLogic Serverfor example, Windows, Solaris, HPUX, Linux, and AIX. On platforms where the performance pack is not available, you must use the Java-based socket reader implementation, which is discussed in detail later in the "Tuning the Java Socket Reader Threads" section.

You can verify if a performance pack is available for your hardware platform on the Certifications pages of the BEA documentation at the following URL:

http://e-docs.bea.com/wls/certifications/certifications/index.html

Even though the performance pack is enabled by default for those platforms that support the native socket multiplexor, as a WebLogic Server administrator, you should always ensure that the performance pack is enabled by following these steps:

1. Start your Administration Server and access the Administration Console.

2. Expand the Servers node in the navigation tree in the left pane.

3. Click the server for which you want to enable or disable the performance pack.

4. In the right pane, click the Configuration, Tuning tab.

5. If the Enable Native IO option checkbox is not selected, enable the option by selecting the check box, as shown in Figure 28.1.

   Figure 28.1. Enable the performance pack for WebLogic Server through the Administration Console.

   

6. Click on the Apply button to save changes to the config.xml file.

   Note

   Enabling the option sets the NativeIOEnabled attribute of the ServerMBean to true and disabling it sets the attribute to false .

7. Restart your WebLogic Server for the changes to take effect.

The performance pack binary ( wlntio.dll file on Windows, a libmuxer.so or libmuxer.sl file depending on the flavor of Unix) should be in your WebLogic Server's PATH . Depending on the platform on which you are running WebLogic Server, the binary is set appropriately in the setWLSEnv.sh script on Unix platforms and setWLSEnv.cmd on the Windows platform. This script is present in the $WL_HOME/server/bin directory of your WebLogic server installation.

With the Native IO option enabled, you will notice the following log statements in your WebLogic Server's log file.

For information on how to use and view your WebLogic Server log files, see "Understanding the WebLogic Server Logs," p. 827 .

For Windows:

 <Dec 5, 2002 11:58:02 PM PST> <Info> <socket> <000406> <NTSocketMuxer was built on Jun 24 2002 17:35:19> <Dec 5, 2002 11:58:02 PM PST> < Info > <socket> <000408> <Allocating 2 NT reader threads> 

For Unix

 <Nov 7, 2002 10:49:39 AM CST> <Info> <Posix Performance Pack> <devsun2> <adminserver> <ListenThread> <system> <> <000000> <System has file descriptor limits of - soft: '1024', hard: '1024'> ####<Nov 7, 2002 10:49:39 AM CST> <Info> <Posix Performance Pack> <devsun2> <adminserver> <ListenThread> <system> <> <000000> <Using effective file descriptor limit of: '1024' open sockets/files.> ####<Nov 7, 2002 10:49:39 AM CST> <Info> <Posix Performance Pack> <devsun2> <adminserver> <ListenThread> <system> <> <000000> <Allocating: '3' POSIX reader threads> 

Hence, by default, when you use the performance pack on the Windows platform, the number of socket reader threads is twice the number of CPUs on that specific server machine. On a Unix platform, the number of socket reader threads is set to 3 by default.

WebLogic clients do not have performance packs and have to rely on the Java socket multiplexor implementation to read data. The following section discusses how you can tune the threads using the Java socket multiplexor implementation.

Tuning the Java Socket Reader Threads

Whenever possible, it is recommended that you use the performance pack provided while running any WebLogic Server instance. This makes for a very efficient native implementation of the socket multiplexor as compared to the pure-Java implementation.

In cases when this is not possible, you have to use the Java socket reader implementation, and configuring the appropriate number of socket reader threads for each WebLogic Server instance and client will improve performance.

The socket reader threads are a part of the default execute queue. The ThreadPoolPercentSocketReaders attribute of the ServerMBean allows you to set the percentage of execute threads from the default queue that will be used for socket reading. The default value for this attribute is 33 (percent) and the range of legal values is 199.

Increasing the value of the ThreadPoolPercentSocketReaders attribute will improve the socket-reading performance of the server because there will be a greater number of execute threads dedicated to socket reading. However, because the socket reader threads are a part of the default execute queue, increasing this value cuts down on the number of threads available for actually executing the tasks.

The value you assign to the ThreadPoolPercentSocketReaders attribute depends on your application type. For example:

• If your application requests are CPU intensive or database intensive and take a longer amount of time to respond, you will need a greater number of actual execute threads than socket reader threads.

• If your application requests can be serviced in a very short time, increasing the socket reader threads will yield a better performance.

To set the number of socket reader threads for WebLogic Server, follow these steps:

1. Start the Administration Server and access the Administration Console.

2. Expand the Server node in the navigation tree in the left pane.

3. Click the server name for which you want to tune the socket reader threads.

4. In the right pane, click the Configuration, Tuning tab.

5. Set the value of the Socket Readers attribute. The value is a percentage of the total number of threads available in the default execute queue.

6. Click the Apply button to apply the changes.

7. Restart the server for the changes to take effect.

Note

Setting the Socket Readers attribute on the server makes sense only when you are using the pure-Java socket multiplexor implementation for your WebLogic Server.

On the client side, you cannot use the performance pack, so you must default to the Java socket multiplexor implementation. If your client is I/O intensive, making several remote calls, then increasing the socket reader count on the client side will improve performance. On the client side, you can do this by setting the value of the ThreadPoolPercentSocketReaders system property. You can also increase the total number of execute threads available on the client side by setting the value of the ThreadPoolSize system property.

You can configure the number of socket reader threads on the client side by specifying the following options in the Java command line for the client:

• -Dweblogic.ThreadPoolSize= value

• -Dweblogic.ThreadPoolPercentSocketReaders=value

WebLogic Server Execute Queues

As mentioned earlier in this chapter, WebLogic Server is a multithreaded application server. The server maintains a pool of Java threads that do all the work. Using multiple threads allows the tasks to be executed in parallel and achieve high performance. These threads in WebLogic Server are called execute threads (or worker threads) because the tasks are executed by them.

The threads within WebLogic Server are partitioned into a set of queues called execute queues . Each queue has the responsibility of executing a specific type of task. The tasks can be application specific such as handling servlet/JSP requests, executing remote EJB methods , and establishing JDBC connections, or they can be server specific tasks such as handling Administration Console requests, handling transaction triggers, running dynamic garbage collection, handling replication requests, and so on. The execute threads are partitioned into execute queues so that one particular task does not take over all the thread resources and cause thread starvation for other types of tasks.

Examples of some of the execute queues in WebLogic Server include

• __weblogic_admin_rmi_queue (10 threads) Requests from a managed server to Administration Server are executed in this queue.

• __weblogic_admin_html_queue (2 threads) The Administration Console requests are executed in this queue.

• _weblogic_dgc_queue (2 threads) Dynamic garbage collection is performed in the threads from this queue.

• The Default queue (15 threads) Unless you have configured a custom execute queue for an application, all application requests, by default, are executed by the threads in this queue.

• Multicast queue (1 thread) The threads in this queue are used for handling multicast communication while using Clustering.

As shown in Figure 28.2, work requests enter WebLogic Server through either the listen thread (listens on port 7001 by default) or the SSL listen thread (listens on port 7002 by default), depending on the protocol used for sending the request. These requests are read by the socket reader threads and then placed on an appropriate execute queue depending on the request type. If no custom execute queues are configured, the requests are placed on the default execute queue.

Figure 28.2. The internals of the WebLogic Server thread pool.

Note

WebLogic Server also allows you to configure custom execute queues on a per-application basis. These queues are discussed later in this chapter in the "Assigning Applications to Specific Execute Queues" section.

As execute threads (worker threads) within the specific queue become free, they pick up the requests from the execute queue, execute the task, and send back the response.

Determining the Appropriate Thread Count for an Execute Queue

WebLogic Server allows you to tune the number of execute threads (worker threads) within the default execute queue. You will need to tune this number if you want to control the degree of concurrency/parallelism possible within the serverfor example, the number of simultaneous operations performed by the applications deployed on WebLogic Server. However, increasing this number does not necessarily guarantee better performance because more threads mean more context switching between threads and more memory usage, which could lead to a drop in performance.

Caution

Increasing the number of execute threads within the default execute queue inappropriately may degrade performance; therefore, you should be very careful while tuning the execute threads.

The optimum number of execute threads in your WebLogic Server's default execute queue, from which you can derive performance value, is influenced by the following factors:

• The number of concurrent requests being handled by the server (throughput)

• The average response time for each request, which is specific to the type of application you have deployed to your server (response time)

• The number of work requests waiting to be executed by the worker threads (queue length)

• The number of idle worker threads during peak usage of your WebLogic Server application (idle threads)

• The number of CPUs and their utilization on your server machine (CPU usage)

• The amount of memory (RAM) dedicated to WebLogic Server on your machine (memory usage)

Taking into account these factors, the following sections describe how to tune your WebLogic Server's execute thread count in the default execute queue.

Step 1: Start the Administration Server

Start your Administration Server. If you are not running the Administration Server in standalone mode, start the managed server(s) where your application(s) is deployed. By default, your WebLogic Server(s) will start with a default thread count of 15.

Step 2: Run and Monitor a Load Test Against Your WebLogic Server Application

To determine the ideal thread count for your default execute queue, you must run a load test against your WebLogic Server application and monitor the factors that can influence the queue's performance, which were mentioned earlier (throughput, response time, queue length, idle threads, CPU usage, and memory usage). This load test must simulate the important functional areas of your application operating at peak usage, which should include access to Enterprise Information Systems, such as database and legacy systems.

You can perform a load test using a variety of performance analysis tools, as listed in Table 28.1. For example, a very easy performance analysis tool to use is the Web Benchmark tool, which you can download for free from BEA's Dev2Dev Web site ( http://dev2dev.bea.com ). Load testing your WebLogic Server application using the Web Benchmark requires only a few configuration steps, as follows :

1. Download the Web Benchmark zip file ( bench.zip ) from the Dev2Dev Web site into a directory on your machinefor example, <drive>:\ benchmark .

2. Unzip the bench.zip file into its current directory.

3. Because the Web Benchmark tool is a Java client, you must include its directory in the CLASSPATH environment variable for your operating system. For example, you can set the CLASSPATH in Windows using the following command:

     set CLASSPATH=<drive>\benchmark;%CLASSPATH%  

After you configure the Web Benchmark tool, you can run a load test against your WebLogic Server application using the following command-line syntax:

 java BenchClient <  #threads>  <  host  > <  port  > "<  URI  >" <  #iterations  > 

where

• #threads The number of concurrent clients you want to simulate connecting with WebLogic Server

• host The IP address or DNS name of the target WebLogic Server to load test

• port The port WebLogic Server is listening on

• URI The virtual mapping, extra path information, and query string of the URL

• #iterations The number of requests each client will perform

For example, the following Web Benchmark command will simulate 15 clients, each performing 5,000 requests to the full URL of http://EINSTEIN:7001/HelloWorldApp/MyServlet :

 java BenchClient 15 EINSTEIN 7001 "/HelloWorldApp/MyServlet" 5000 

Note

The Web Benchmark tool allows you to hit Web pages in your application with multiple threads and multiple iterations. However, it does not support the use of POST requests or session tracking.

The output you get from running the Web Benchmark indicates the latency of each thread (client), listed from the shortest to the longest amount of time. The output of the Web Benchmark tool will be examined later in this section.

To monitor your load test, you can again use a performance analysis tool with built-in monitoring capabilities, or you can use the Administration Console's monitoring capability, which should be sufficient for monitoring the influential performance tuning factors of the execute threads in the default execute queue.

To monitor the WebLogic Server environment, in conjunction with your performance analysis tool, you can also use the following system performance tools:

• The Windows Performance Tool, which can be used to monitor and capture the performance statistics on many system characteristics, including the following:

  • CPU utilization

  • Memory utilization

  • Operating system and application thread usage

  • Active processes running on the system

  • Network activity

  • I/O statistics

• Unix system performance monitoring utilities, such as the following:

  • sar System activity (Solaris)

  • mpstat Per-processor (Solaris)

  • vmstat Virtual memory statistics

  • netstat Network statistics

  • iostat I/O statistics

To showcase how you can use the Web Benchmark tool in conjunction with the Administration Console to load test and monitor the factors that can influence the execute queue's performance, we'll use a very simple example that will simulate 15 clients, each performing 10,000 requests to a simple URL. You can adjust the syntax of the Web Benchmark tool command according to your own WebLogic Server application pages.

Note

With the evaluation version of WebLogic Server, the maximum number of clients you can concurrently simulate is 15.

To monitor the load test using the Administration Console, follow these steps:

1. Access the Administration Console.

2. Click the Servers node in the left pane to display the servers configured in your domain.

3. Click the name of the server instance that you want to monitor.

4. Select the Monitoring, Performance tab, as shown in Figure 28.3.

   Figure 28.3. The Performance monitoring pane of the Administration Console.

   

From the Performance tab of the Administration Console, as shown in Figure 28.3, you can monitor the following factors during the load test:

• The number of idle threads assigned to the queue

• The time that the longest waiting request was placed in the queue

• The number of requests that have been processed by this queue

• The number of waiting requests in the queue

• The current amount of free memory (in bytes) in the JVM heap

Tip

You can adjust the refresh rate of the Administration Console and the polling interval ( milliseconds ) for graph data from the Console, Preferences tab. The polling interval will affect your displayed data because it will display the results based on your polling interval.

To monitor the CPU utilization doing the load test, you can use either the Windows Performance Monitoring tool or an appropriate Unix utility, such as mpstat .

Now that you know how to run and monitor a simple load test, the next step is to actually start the load testing and monitoring activities, which begins with running the Web Benchmark command. The command used for this simple showcase is

 java BenchClient 15 EINSTEIN 7001 "/HelloWorldApp/MyServlet" 10000 

Note

Because the purpose of this test is to simulate actual client interaction with WebLogic Server, it should be run from an actual client machine.

As the Web Benchmark simulates your client load test, you can monitor the activity of your test via the Administration Console and other system monitoring tools. After the Web Benchmark completes its load test on your WebLogic Server application, it will display the results of the latency of the client requests to complete their iterations in the shortest to the longest time, as shown in Figure 28.4.

Figure 28.4. The results of running the Web Benchmark tool in a simulated load test.

The results from the Administration Console, as shown in Figure 28.5, also provide valuable information, as follows:

Figure 28.5. The results of running the Web Benchmark tool through the Administration Console.

• The peak number of idle threads during the load test. However, you will need to refresh the Administration Console at multiple times during the load test to capture this value. From the test, there were a maximum of four idle threads.

• The peak number of requests executed at the polling interval of the performance graph. From the test, there was a peak of 1,237 requests executed.

• The peak number of requests waiting in the execute queue. From the test, there was a peak of 11 requests waiting to be executed in the queue.

• The maximum memory utilization during the test.

A Windows Performance Monitor tool also used during the load test indicated a maximum of 100% and a low of 84% CPU utilization on a dual CPU machine, as shown in Figure 28.6.

Figure 28.6. The CPU utilization results of running the Web Benchmark tool through the Windows Performance Monitoring tool.

You should perform your load test a number of times to a point where your results are relatively consistent, at which point you can analyze them.

In general, if the thread count is set too low, you will see the following results from your load test:

• The CPU is waiting to do work, but there is work that can be performed.

• The CPU utilization never consistently stays within the 85% to 95% optimal utilization range.

• There is a high percentage of requests waiting to be executed (queue length) in comparison to the throughput of the queue. This is dependent on the type of application and the functional area of that application being tested .

• The latency of the client requests is high based on the performance requirements for the functional areas of the WebLogic Server application being load tested.

In general, if the thread count is set too high, you will see the following results from your load test:

• The CPU utilization is consistently at 100% utilization.

• The latency of the client requests is still high based on the performance requirements for the functional areas of the WebLogic Server application being load tested.

• The throughput has greatly increased, but the peak number of requests in the queue remains relatively constant, and there is also an increase in the memory utilization.

The hard and fast way to validate whether you are running your WebLogic Server with high or low thread count is to increase the thread count and compare your results. If there is an increase in performance, you need to repeat your tests by incrementing the thread count by a small amount (two to five threads only) until you reach a point where you see a performance degradation in the throughput and latency of the client requests. This will narrow your execute thread count to a small range for fine-tuning. The following section describes how you can modify the thread count in the default execute queue.

However, if you increase your thread count and see an immediate performance degradation, you will have to take the same iterative test approach, but this time by decreasing the thread count by a small amount until you see a performance increase. This will also narrow your execute thread count to a small range for fine tuning.

Step 3: Modifying the Thread Count in the Default Execute Queue

You can modify the number of execute threads in the default execute queue using the ThreadCount attribute of the ServerMBean. To set this attribute via the Administration Console, follow these steps:

1. Start the Administration Server and access the Administration Console.

2. Click the Servers node in the left pane to display the servers configured in your domain.

3. Click the name of the server instance that contains the default execute queue you want to configure.

4. Select the Monitoring, General tab in the right pane.

5. Click the Monitor All Active Queues text link to display the active execute queues that the selected server uses, for example, as shown in Figure 28.7.

   Figure 28.7. The active execute queues in a default WebLogic Server installation.

   

   Tip

   You can use the screen shown in Figure 28.7 for monitoring the thread activity in an execute queue. However, the refresh rates for an Administration Console screen are much longer than the data presented via the graph (refer to Figure 28.5).

6. Click the Configure Execute Queue text link to display the execute queues that you can modify.

   Note

   You can modify only the default execute queue for a server or a user -defined execute queue.

7. In the table of configured execute queues, click the name of the default execute queue to display the Execute Queue Configuration tab, as shown in Figure 28.8.

   Figure 28.8. The default execute queue's Configuration tab.

   

8. From the displayed Execute Queue Configuration tab, increase or decrease the default thread count of 15 by a small amount (2 to 5 threads).

9. Click Apply to apply your changes.

10. Shut down and restart the selected server to enable the new thread count settings for the default execute queue.

Tuning Execute Queues for Overflow Conditions

The socket reader threads in WebLogic Server read requests and place them on the appropriate execute queue. The length of this execute queue will increase if the requests come in at a rate faster than they are processed. In such situations, the response time for the requests will be slower. WebLogic Server allows you to configure attributes for the execute queue, which helps in detecting such conditions.

The server detects an overflow condition if the length of the execute queue reaches a user-defined percentage of its maximum size. When this threshold is reached, the server's state is changed to Warning, and depending on other attributes set for the queue, additional execute threads can be allocated to the queue. These additional execute threads then pick up the requests from the execute queue and thus help in reducing the queue length to an acceptable limit.

To detect the overflow conditions and handle them automatically, follow these steps:

1. Start the Administration Server and access the Administration Console.

2. Click the Servers node in the left pane to display the servers configured in your domain.

3. Click the name of the server instance that contains the execute queue you want to configure.

4. Select the Monitoring, General tab in the right pane.

5. Click the Monitor All Active Queues text link to display the active execute queues that the selected server uses, for example, as shown in Figure 28.7.

6. Click the Configure Execute Queue text link to display the execute queues that you can modify.

   Note

   You can configure these attributes only for the default queue or any custom execute queue that you have configured. Parameters for the WebLogic-specific execute queues (such as __weblogic_admin_rmi_queue ) cannot be configured.

7. In the table of configured execute queues, click the name of the default execute queue to display the Execute Queue Configuration tab (refer to Figure 28.8).

8. From the displayed Execute Queue Configuration tab modify the following attributes:

   • The Queue Length parameter indicates the maximum number of requests that can be in the execute queue. The maximum size is 65535 and the minimum size is 256. In most cases, you don't need to modify the Queue Length parameter.

   • Set the Queue Length Threshold Percent attribute. The default is set to 90%. This threshold attribute indicates a percentage of the queue length size and is monitored by the server for detecting overflow conditions. You can set this percentage to any value between 1 and 99. Queue length sizes below the threshold percentage are considered normal. If the queue length size reaches the percentage value you've set, an overflow condition is detected and handled by the server.

   • Set the Threads Increase parameter to the number of threads that the WebLogic Server should allocate to the execute queue when an overflow condition is detected. The server uses this attribute to automatically handle an overflow condition. The additional threads allocated will help in reducing the execute queue length. Enter for the number of threads if you do not want the server to automatically allocate execute threads to the queue.

     Threads added to an execute queue after the detection of an overflow condition will not be removed even after the queue length drops back to an acceptable limit. The number of threads in the queue thus stands increased after an overflow condition is detected and if you have set a non-zero value for the Threads Increase parameter.

   • Set the Threads Maximum attribute to prevent the server from increasing threads to a very high number in response to an overflow condition. On detection of an overflow condition, the server will increase the threads in the execute queue by a number set in the Threads Increase parameter. This may hamper performance, so it is important to control the maximum number of threads using this parameter. The default value is 400.

   • Set the Threads Minimum attribute to specify the minimum number of threads that the server should maintain in the execute queue to prevent an unnecessary overflow condition. The default value is 5.

9. Click the Apply button to apply the changes.

10. Shutdown and restart the server for these changes to take effect.

The server prints the following message in the log file when it detects an overflow condition and adds ThreadsIncrease number of threads to the execute queue:

 <Dec 5, 2002 4:59:08 PM PST> <Warning> <WebLogicServer> <000333> <Queue usage is greater than QueueLengthThresholdPercent "5%" of the maximum queue size. We will try to allocate ThreadsIncrease "2" thread(s) to help.> 

Determining Stuck Threads

When tasks are put on an execute queue by the socket reader threads, the idle execute threads pick up the task, execute it, and send back the response. Idle execute threads should always be available within the execute queue to perform this task. If no idle threads are available, the tasks will not be performed. This is a critical condition because the requests will go unserved.

WebLogic Server automatically detects conditions in which an execute thread takes more than an acceptable amount of time to complete its task. The server marks such threads as stuck . If all the threads within an execute queue become stuck, the server changes its state to Critical or Warning, depending on the execute queue type. The server's health state changes to Critical if all the threads in the "default" queue are determined to be stuck. The server's health changes to Warning if all the threads in the weblogic_admin_html_queue , weblogic_admin_rmi_queue , or a user-defined execute queue become stuck.

Note

If a WebLogic Server's health state changes to Critical, it is very likely the server's health state will be labeled "failed," at which point if you are using the Node Manager, it can be used to shut down and restart the "failed" WebLogic Server. However, a change of health state to Warning does not necessarily imply WebLogic Server will be labeled "failed."

For information on WebLogic Server Self-Health monitoring capabilities, see "Monitoring the Health of Your WebLogic Domain Using the Node Manager," p. 824 .

To configure the detection of stuck threads in the server, follow these steps:

1. Start the Administration Server, if it is not already running, and access the Administration Console.

2. Expand the Servers node in the navigation tree in the left pane.

3. Click the server name for which you want to configure the Stuck Thread detection parameters.

4. Select the Configuration, Tuning tab in the right pane, as shown in Figure 28.9.

   Figure 28.9. Set attributes to determine Stuck Threads in WebLogic Server using the Administration Console.

   

5. Modify the following attributes in the Configuration, Tuning tab as necessary to tune thread detection behavior for your WebLogic Server:

   • Set the Stuck Thread Max Time attribute. This is the maximum amount of time that an execute thread can be busy. If it exceeds this time, the server marks the execute thread as stuck. The default value for this parameter is 600 seconds. Set the attribute to a value greater than the maximum acceptable time that your server can take to execute a request.

   • Set the Stuck Thread Timer Interval attribute to a time in seconds that the server should periodically check for stuck threads. The default value is 600 seconds, which means that the server will check at 10-minute intervals to find threads that have been working for longer than the Stuck Thread Max Time period.

6. Click the Apply button to apply the changes.

7. Shut down and restart the server for the changes to take effect.

You can verify whether your WebLogic Server is detecting stuck threads by reviewing your WebLogic Server log file. For example, the server prints the following message in the log file when it detects a stuck thread:

 <Dec 5, 2002 4:56:52 PM PST> <Warning> <WebLogicServer> <000337> <ExecuteThread: '10' for queue: 'default' has been busy for "38" seconds working on the request "Http Request: /sleep.jsp", which is more than the configured time (StuckThreadMaxTime) of "30" seconds.> 

WebLogic Server also prints the following message in the log file when a thread marked Stuck by the server completes its work and becomes idle:

 <Dec 5, 2002 7:50:48 PM PST> <Info> <WebLogicServer> <000339> <ExecuteThread: '27' for queue: 'default' has become "unstuck".> 

Configuring Custom Execute Queues

WebLogic Server has a default queue to handle all the application-specific requests. However, it also allows you to configure execute queues on a per-application basis.

You may want to configure these additional execute queues to achieve the following advantages:

• Optimize the performance of critical applications In the default scenario, when you have not configured any custom execute queues, all the applications deployed on WebLogic Server use the execute threads in the default queue to do their work. Because all the applications share the same default queue, it is possible that a critical application will be starved of threads because they are being used by some other applications. To prevent this thread starvation for critical applications, you can configure an execute queue dedicated to them. This solution always guarantees a fixed number of threads for the critical application.

• Throttle the performance on nonessential applications For applications that are nonessential but have a potential to take over server resources such as threads and memory, assigning a dedicated execute queue can help limit the resource starvation that such applications can cause to other applications on the same server.

• Remedy deadlocks in threads Configuring a dedicated execute queue can help prevent out-of-threads deadlock. Consider an application that, as a part of its task, has to talk to another subsystem on the same serverfor example, a servlet calling remote EJB methods. If all the execute threads are processing servlet requests, there will be no threads to process the EJB methods. Because EJB methods are a part of the task, you see an out-of-threads deadlock for the EJBs. You can prevent this problem by assigning the servlet a separate execute queue so that it does not take over all the thread resources from the EJBs.

You need to be aware, however, that unnecessarily configuring custom execute queues for applications wastes resources. You can have a condition in which execute threads in a particular custom queue are idle, while there are no idle threads in another application-specific execute queue. In this case, the idle threads in one queue are wasted resources because they could have been used by the other application if the queues were not partitioned.

To configure custom execute queues using the Administration Console, follow these steps:

1. Start the Administration Server and access the Administration Console.

2. Click the Servers node in the left pane to display the servers configured in your domain.

3. Click the name of the server instance where you want to configure a new custom execute queue.

4. Select the Monitoring, General tab in the right pane.

5. Click the Monitor All Active Queues text link to display the active execute queues that the selected server uses, for example, as shown in Figure 28.7.

6. Click the Configure Execute Queue text link to display the execute queues that you can modify.

7. Click the Configure a New Execute Queue link to show the Execute Queue Configuration tab, as shown in Figure 28.10.

   Figure 28.10. Configure a new execute queue using the Administration Console.

   

8. In the Execute Queue Configuration tab, modify the following attributes or accept the system defaults:

   • Enter a name for your new execute thread.

   • Set the Thread Count attribute to configure the number of execute threads in the custom execute queue.

   Set other parameters as discussed in the "Tuning Execute Queues for Overflow Conditions" section earlier in this chapter.

9. Click the Apply button to apply the changes. The changes are persisted to the config.xml file.

10. Restart the server for the changes to take effect.

Assigning Applications to Specific Execute Queues

In the previous section, we discussed the advantages of assigning dedicated execute queues to applications. In this section, we will discuss how applications such as servlets, JSPs, and EJBs can be assigned to dedicated execute queues.

Assigning Servlets and JSPs to Execute Queues

To assign servlets and JSPs to a dedicated execute queue that has been already configured in the server, you can use the <wl-dispatch-policy> parameter to specify the queue in which the servlet or JSP will be executed. This parameter is specified as an initialization parameter in the web.xml deployment descriptor of the Web application containing the servlets and JSPs.

The following snippet from the web.xml file assigns a dedicated execute queue called CriticalAppQueue to the SnoopServlet and the critical.jsp file. Any requests for these Web files will be executed by threads within the CriticalAppQueue :

 <servlet> <servlet-name>SnoopServlet</servlet-name> <jsp-file>/myapp/critical.jsp</jsp-file> <init-param> <param-name>wl-dispatch-policy</param-name> <param-value>CriticalAppQueue</param-value> </init-param> </servlet> 

Assigning RMI Objects and EJBs to Execute Queues

To assign an RMI object or EJB to a dedicated execute queue that has already been configured in the server, you can use the -dispatchpolicy option while using the rmic and ejbc compilers. Use the rmic compiler as follows:

 java weblogic.rmic -dispatchPolicy CriticalAppQueue examples.HelloImpl 

If the RMI implementation class is compiled as shown, any requests for the HelloImpl remote object will be executed on the server in an execute thread from the CriticalAppQueue .

Use the ejbc compiler as follows:

 java weblogic.ejbc -dispatchPolicy CriticalAppQueue std_AccountBean.jar AccountBean.jar 

If the EJB AccountBean is compiled as shown, any requests for the remote methods of the AccountBean will be executed on the server in an execute thread from the CriticalAppQueue .

Monitoring Execute Queues and Execute Threads

To monitor the execute queues, execute threads, and their attributes in WebLogic Server, follow these steps:

1. Start the Administration Server, if it is not already running, and access the Administration Console.

2. Click the Servers node in the left pane to display the servers configured in your domain.

3. Click the name of the server instance that contains the execute queue you want to monitor.

4. Select the Monitoring, General tab in the right pane.

5. Click the Monitor All Active Queues text link to display the active execute queues that the selected server uses, for example, as shown in Figure 28.7. You will see a listing of all the active execute queues in the server and each queue's attributesnumber of threads, number of idle threads, throughput, and so on.

6. Click a particular queue name in the table listing to view all the execute threads within that execute queue. For example, Figure 28.11 shows a listing of all execute threads within the default execute queue.

   Figure 28.11. A list of all the execute threads within the default execute queue.

   

Tuning the Connection Backlog

The AcceptBackLog parameter allows you to configure the number of TCP connection requests that can be buffered in a wait queue before being accepted by the server. WebLogic Server, by default, listens on two ports: the plain-text port (default 7001) and the SSL port (default 7002). If work requests come to these ports at a rate faster than they are being processed, they will be buffered in a queue whose size is fixed by this parameter.

You will need to tune this parameter if either the connections are being dropped silently on the client side or the clients are receiving connection refused messages when trying to connect to WebLogic Server.

The default value of this parameter is 50, which means that a maximum of 50 connection requests can be buffered at any given time. If the queue becomes full, additional connection requests will be dropped and not accepted by the server.

Caution

Setting this parameter to 0 may prevent the server from accepting any connections on some operating systems.

To set this parameter, follow these steps:

1. Start the Administration Server and access the Administration Console.

2. Expand the Servers node in the navigation tree in the left pane.

3. Click the name of the server instance that you want to configure.

4. Select the Connection, Tuning tab in the right pane, as shown in Figure 28.12.

   Figure 28.12. Setting the AcceptBackLog parameter through the Administration Console.

   

5. Set the AcceptBackLog parameter as necessary to tune how many TCP connections can be buffered in a wait queue.

6. Click the Apply button to persist the value to the config.xml file.

7. Shut down and restart WebLogic Server for the changes to take effect.

To tune this parameter, start a realistic load test on the server. If you see connections being refused on the client side, increase the AcceptBackLog value by 25% and retest. Continue increasing the parameter by 25% until the queue size is large enough to buffer the connection requests generated by your load test.