Tuning the Default Execute Queue Threads

The value of the

ThreadCount

attribute of an

ExecuteQueue

element in the

config.xml

file equals the number of simultaneous operations that can be performed by applications that use the execute queue. As work enters an instance of WebLogic Server, it is placed in an execute queue. This work is then assigned to a thread that does the work on it. Threads consume resources, so handle this attribute with care—you can degrade performance by increasing the value unnecessarily.

By default, a new WebLogic Server instance is configured with a development mode execute queue, weblogic.kernel.default, that contains 15 threads. In addition, WebLogic Server provides two other pre-configured queues:

• weblogic.admin.HTTP—Available only on Administration Servers, this queue is reserved for communicating with the Administration Console; you cannot reconfigure it.
• weblogic.admin.RMI—Both Administration Servers and Managed Servers have this queue; it is reserved for administrative traffic; you cannot reconfigure it.

Unless you configure additional execute queues, and assign applications to them, Web applications and RMI objects use weblogic.kernel.default.

Note: If native performance packs are not being used for your platform, you may need to tune the default number of execute queue threads and the percentage of threads that act as socket readers to achieve optimal performance. For more information, see Allocating Execute Threads to Act as Socket Readers.

Should You Modify the Default Thread Count?

Adding more threads to the default execute queue does not necessarily imply that you can process more work. Even if you add more threads, you are still limited by the power of your processor. Because threads consume memory, you can degrade performance by increasing the value of the ThreadCount attribute unnecessarily. A high execute thread count causes more memory to be used and increases context switching, which can degrade performance.

The value of the ThreadCount attribute depends very much on the type of work your application does. For example, if your client application is thin and does a lot of its work through remote invocation, that client application will spend more time connected — and thus will require a higher thread count — than a client application that does a lot of client-side processing.

If you do not need to use more than 15 threads (the development default) or 25 threads (the production default) for your work, do not change the value of this attribute. As a general rule, if your application makes database calls that take a long time to return, you will need more execute threads than an application that makes calls that are short and turn over very rapidly. For the latter case, using a smaller number of execute threads could improve performance.

Scenarios for Modifying the Default Thread Count

To determine the ideal thread count for an execute queue, monitor the queue’s throughput while all applications in the queue are operating at maximum load. Increase the number of threads in the queue and repeat the load test until you reach the optimal throughput for the queue. (At some point, increasing the number of threads will lead to enough context switching that the throughput for the queue begins to decrease.)

Note: The WebLogic Server Administration Console displays the cumulative throughput for all of a server’s execute queues. To access this throughput value, follow steps 1-6 in Modifying the Default Thread Count.

Table 4-3 shows default scenarios for adjusting available threads in relation to the number of CPUs available in the WebLogic Server domain. These scenarios also assume that WebLogic Server is running under maximum load, and that all thread requests are satisfied by using the default execute queue. If you configure additional execute queues and assign applications to specific queues, monitor results on a pool-by-pool basis.

Table 4-3 Scenarios for Modifying the Default Thread Count When…	Results	Do This:
Thread Count < number of CPUs	Your thread count is too low if:                        CPU is waiting to do work, but there is work that could be done.                         Cannot get 100 percent CPU utilization rate.	Increase the thread count.
Thread Count = number of CPUs	Theoretically ideal, but the CPUs are still under-utilized.	Increase the thread count.
Thread Count > number of CPUs (by a moderate number of threads)	Practically ideal, with a moderate amount of context switching and a high CPU utilization rate.	Tune the moderate number of threads and compare performance results.
Thread Count > number of CPUs (by a large number of threads)	Too much context switching, which can lead to significant performance degradation. Your performance may increase as you decrease the number of threads.	Reduce the number of threads so that it equals the number of CPUs, and then add only the number of "stuck" threads that you have determined. For example, if you have four processors, then four threads can be running concurrently with the number of stuck threads. So, you want the execute threads to be 4 + the number of stuck threads. To determine the amount of stuck threads, see Tuning the Execute Thread Detection Behavior. Note: This recommendation is highly application-dependent. For instance, the length of time the application might block threads can invalidate the formula.

Modifying the Default Thread Count

To modify the default execute queue thread count using the Administration Console:

1. Start the Administration Server if it is not already running.

1. Access the Administration Console for the domain.

1. Expand the Servers node in the left pane to display the servers configured in your domain.

1. Right-click the name of the server instance that contains the execute queue you want to configure, and then select View Execute Queues on the pop-up menu to display a table of execute queues that can be modified.

Note: You can only modify the default execute queue for the server or a user-defined execute queue.

1. In the Name column, click directly on the default execute queue name to display the Configuration tab for modifying execute queues.

1. Locate the Thread Count value and increase or decrease it, as appropriate.

1. Click Apply to save your changes.

1. Reboot the selected server to enable the new execute queue settings.

Assigning Applications to Execute Queues

Although you can configure the default execute queue to supply the optimal number threads for all WebLogic Server applications, configuring multiple execute queues can provide additional control for key applications. By using multiple execute queues, you can guarantee that selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Server. See Using Execute Queues to Control Thread Usage for more information on assigning applications to configured execute queues.

Allocating Execute Threads to Act as Socket Readers

For best socket performance, BEA recommends that you use the native socket reader implementation, rather than the pure-Java implementation, on machines that host WebLogic Server instances (see Using WebLogic Server "Native IO" Performance Packs). However, if you must use the pure-Java socket reader implementation for host machines, you can still improve the performance of socket communication by configuring the proper number of execute threads to act as socket reader threads for each server instance and client machine.

The ThreadPoolPercentSocketReaders attribute sets the maximum percentage of execute threads that are set to read messages from a socket. The optimal value for this attribute is application-specific. The default value is 33, and the valid range is 1-99.

Allocating execute threads to act as socket reader threads increases the speed and the ability of the server to accept client requests. It is essential to balance the number of execute threads that are devoted to reading messages from a socket and those threads that perform the actual execution of tasks in the server.

Setting the Number of Socket Reader Threads For a Server Instance

To use the Administration Console to set the maximum percentage of execute threads that read messages from a socket:

1. Start the Administration Server if it is not already running.

1. Access the Administration Console for the domain.

1. Expand the Servers node in the left pane to display the servers configured in your domain.

1. Click the name of the server that you want to configure.

1. Select the Configuration —> Tuning tab.

1. Edit the percentage of Java reader threads in the Socket Readers attribute field. The number of Java socket readers is computed as a percentage of the number of total execute threads (as shown in the Thread Count field for the Execute Queue).

1. Apply the changes.

Setting the Number of Socket Reader Threads on Client Machines

On client machines, you can configure the number of available socket reader threads in the JVM that runs the client. Specify the socket readers by defining the following parameters in the java command line for the client:

-Dweblogic.ThreadPoolSize=value
-Dweblogic.ThreadPoolPercentSocketReaders=value

Tuning Execute Queues for Overflow Conditions

You can configure WebLogic Server to detect and optionally address potential overflow conditions in the default execute queue or any user-defined execute queue. WebLogic Server considers a queue to have a possible overflow condition when its current size reaches a user-defined percentage of its maximum size. When this threshold is reached, the server changes its health state to "warning" and can optionally allocate additional threads to perform the outstanding work in the queue, thereby reducing its size.

To automatically detect and address overflow conditions in a queue, you configure the following items:

• The threshold at which the server indicates an overflow condition. This value is set as a percentage of the configured size of the execute queue (the QueueLength value).
• The number of threads to add to the execute queue when an overflow condition is detected. These additional threads work to reduce the size of the queue and reduce the size of the queue to its normal operating size.
• The minimum and maximum number of threads available to the queue. In particular, setting the maximum number of threads prevents the server from assigning an overly high thread count in response to overload conditions.

To tune an execute queue using the WebLogic Server Administration Console:

1. Start the Administration Server if it is not already running.

1. Access the Administration Console for the domain.

1. Expand the Servers node in the left pane to display the servers configured in your domain.

1. Right-click the name of the server instance that contains the execute queue you want to configure, and then select View Execute Queues from the pop-up menu to display a table of execute queues that can be modified.

Note: You can only modify the default execute queue for the server or a user-defined execute queue.

1. In the Name column, directly click the default execute queue name (or the user-defined execute queue) that you want to configure.

1. On the execute queue Configuration tab, specify how the server instance should detect an overflow condition for the selected queue by modifying the following attributes:
   • Queue Length: Specifies the maximum number of simultaneous requests that the server can hold in the queue. The default of 65536 requests represents a very large number of requests; outstanding requests in the queue should rarely, if ever reach this maximum value. Always leave the Queue Length at the default value of 65536 entries.
   • Queue Length Threshold Percent: The percentage (from 1-99) of the Queue Length size that can be reached before the server indicates an overflow condition for the queue. All actual queue length sizes below the threshold percentage are considered normal; sizes above the threshold percentage indicate an overflow. By default, the Queue Length Threshold Percent is set to 90 percent.
   • Thread Priority: The priority of the threads associated with the queue. By default, the Thread Priority is set to 5.

1. To specify how this server should address an overflow condition for the selected queue, modify the following attribute:
   • Threads Increase: The number of threads WebLogic Server should add to this execute queue when it detects an overflow condition. If you specify zero threads (the default), the server changes its health state to "warning" in response to an overflow condition in the execute queue, but it does not allocate additional threads to reduce the workload.

1. To fine-tune the variable thread count of this execute queue, modify the following attributes:
   • Threads Minimum: The minimum number of threads that WebLogic Server should maintain in this execute queue to prevent unnecessary overflow conditions. By default, the Threads Minimum is set to 5.
   • Threads Maximum: The maximum number of threads that this execute queue can have; this value prevents WebLogic Server from creating an overly high thread count in the queue in response to continual overflow conditions. By default, the Threads Maximum is set to 400.

1. Click Apply to save your changes.

1. Reboot the selected server to enable the new execute queue settings.

Tuning the Execute Thread Detection Behavior

WebLogic Server automatically detects when a thread in an execute queue becomes "stuck." Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. If all threads in an execute queue become stuck, the server changes its health state to either "warning" or "critical" depending on the execute queue:

• If all threads in the default queue become stuck, the server changes its health state to "critical." (You can set up the Node Manager application to automatically shut down and restart servers in the critical health state. For more information, see "Node Manager Capabilities" in Configuring and Managing WebLogic Server.)
• If all threads in weblogic.admin.HTTP, weblogic.admin.RMI, or a user-defined execute queue become stuck, the server changes its health state to "warning."

WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server’s thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads.

Note: Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the "warning" and "critical" health states when all threads in a particular execute queue become stuck.

To configure WebLogic Server stuck thread detection behavior:

1. Start the Administration Server if it is not already running.

1. Access the Administration Console for the domain.

1. Expand the Servers node in the left pane to display the servers configured in your domain.

1. Click the name of the server instance that you want to modify for improved stuck thread detection.

Note: You configure stuck thread detection parameters on a per-server basis, rather than on a per-execute queue basis.

1. Select the Configuration —> Tuning tab in the right pane.

1. Modify the following attributes as necessary to tune thread detection behavior for the server:
   • Stuck Thread Max Time: Enter the number of seconds, that a thread must be continually working before this server diagnoses the thread as being stuck. By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
   • Stuck Thread Timer Interval: Enter the number of seconds, after which WebLogic Server periodically scans threads to see if they have been continually working for the length of time specified by Stuck Thread Max Time. By default, WebLogic Server sets this interval to 600 seconds.

1. Click Apply to save your changes.

1. Reboot the server to use the new settings.