Shulou(Shulou.com)05/31 Report--

This article is to share with you the solution to the exhaustion of Logic execution threads. Xiaobian thinks it is quite practical, so share it with you to learn. I hope you can gain something after reading this article. Let's not say much. Let's take a look at it together with Xiaobian.

Solution to Thread Exhaustion in Logic Execution

Running out of execution threads

1.1 Overview

More Related Knowledge:

Common scripts and operation and maintenance skills in the daily life of WebLogic cluster management Graphical installation and silent installation

When the LOGY server runs out of threads and no longer responds to subsequent requests, a pending phenomenon occurs.

There are many reasons for Logic thread exhaustion. Generally speaking, thread exhaustion is due to threads competing for some resources. When a resource is missing, for example, the database JDBC connection pool reaches the maximum number of connections, it will cause subsequent business threads to be stuck waiting to obtain JDBC connections, and finally cause the server to fail to respond to subsequent service requests.

1.2 Common thread exhaustion causes

The most common reasons for running out of threads on a LOGY server are as follows:

Bug in JDK itself

Bugs in open source software used in business systems

The host hardware reaches the upper bottleneck and cannot bear more load.

The database JDBC connection pool is exhausted, and subsequent business threads cannot obtain an available JDBC connection and are always in a waiting state.

Application synchronization calls. For example, service thread A locks resource 1 during execution, and other service threads have to wait indefinitely for the lock of resource 1 to be released due to synchronous invocation.

The business system does not use a ipped thread pool, but a custom thread pool. When the custom thread pool is fully occupied, there are no threads available for new work.

Threads are waiting for RJVM, RMI responses

Application configuration is unreasonable, such as setting the wrong JSP PageCheckSeconds parameter, etc.

1.3 Judgement conditions

Suspect <$Logic Server thread exhaustion when:

Server does not respond to new requests

request timeout

Request processing takes longer and longer (the end result may be a hang)

Typically, a server hang does not appear as a server crash, but a server hang can be followed by a crash.

1.4 Collection of information

When a server hangs, first ping the server using java weblogic.Admin t3://server:port PING. If the server is able to respond to this ping, it is likely that the thread pool used by the application is exhausted rather than the server itself.

Logic Server uses the Default thread pool to respond to client service requests prior to version 8.1 and after version 9.0 to respond to client service requests. These are the thread pools that should be checked when thread exhaustion occurs. Usually, thread pool runtime information is analyzed by collecting Thread Dump of JVM. bold style

Linux / Unix system

Kill -3 can be used by any version of to create Thread Dump needed to diagnose problems. Be sure to perform this several times on each server, approximately 3 to 5 seconds apart, to help diagnose deadlock problems.

Thread Dump information can be collected directly by using JDK's jstack command in Software Development Kit version 9.2 or later.

Windows operating system

- can be used in any version of LOGY to create the Thread Dump needed to diagnose problems. Be sure to perform this several times on each server, approximately 3 to 5 seconds apart, to help diagnose deadlock problems.

Thread Dump information can be collected directly by using JDK's jstack command in Software Development Kit version 9.2 or later.

1.5 Implementation steps

Collect basic information related to operating system and Running Logic

Collect thread exhaustion time period Wolves Run Log

Thread Dump

Quickly analyze Thread Dump on the server to initially locate thread exhaustion issues

Restart threads to deplete the WebLogic server and make it serve the outside world as soon as possible

Download the collected Thread Dump locally for detailed offline analysis

1.6 Analysis of Thread Dump

In addition to the execution thread pool used by the business system, LOGY also includes ListenThread thread pool, Socket Reader thread pool, and custom thread pool. Because LOGY software is very complex enterprise software, we need to focus not only on the business execution thread pool but also on other thread pools when analyzing it. A simple illustration of the Logic thread pool is as follows:

Thread dump is an image of the running state of all threads contained in the current JVM, including:

Stack trace for each thread

Current state of thread

Thread Name

Thread number (tid) and corresponding native thread id (nid) and other information

The most useful way to analyze Thread Exhaustion in a LOGY server is to analyze a series of Thread Dump. A series of Thread dumps (typically three or more Thread dumps every three to five seconds) helps analyze state changes or missing changes in each thread as it moves from one Thread Dump to another. After a server hangs due to thread exhaustion on a LOGY server, the Thread Dump of the server generally shows little change in thread state from the first Thread Dump to the last Thread Dump.

View content in Thread Dump

All requests go through ListenThread into the LOGY Server. If ListenThread is missing, no work can be received and therefore no work can be completed. Confirm ListenThread exists in Thread Dump. ListenThread should be in the socketAccept method. The following example illustrates the form of a Listen Thread.

The Socket Reader thread accepts incoming requests from the listening thread queue and places the request in the execution thread queue. If there is no Socket Reader thread in Thread Dump, there is an error somewhere that causes the Socket Reader thread to disappear. There should always be at least three Socket Reader threads. One Socket Reader thread is typically used for polling functions, and the other two are used to process requests. Here is a Socket Reader thread from a Thread Dump example.

Typically, business code uses the default execution thread pool of LOGY, named "Default" before LOGY 8.1 and "weblogic.kernel.Default(self-tuning)" after 9.x. View the state of the thread execution, mainly by looking at the value of the state in the thread dump. At the same time, to locate problems faster, the WebLogic server marks execution threads that run longer than 600 seconds as "STUCK" status, as follows:

Thread status in java

In Thread Dump, the various states of Java threads are as follows:

state

description

NEW

Initial state. The thread has just been created, and the start() method has not been called.

RUNNABLE

Operational status. Indicates that the thread is executing in the Java virtual machine, but may be waiting for other resources of the operating system, such as the CPU

BLOCKED

Blocking condition. Indicates that the thread is waiting for a monitor lock. Indicates that the thread is waiting to acquire a monitor lock in order to enter a synchronous method or block of synchronous code, or may be awakened from the wait () method and waiting to enter a synchronous method or block of synchronous code again

WAITING

Waiting state. Indicates that the current thread needs to wait for other threads to perform some special operation, such as the current thread calls the a.wait() method, it is waiting for other threads to call the a.notify or a.notifyAll method; if the current thread calls threada.join(), then it is waiting for threada execution to complete.

TIMED_WAITING

Time out waiting. Unlike WAITING, this state has a timeout.

TERMINATED

Terminated state, indicating that the current thread has completed execution

For a detailed description of thread state, please refer to the Thread internal class of Java source code: State.

The difference between WAITING and TIMED_WAITING in thread state is as follows:

Calling the following three methods will enter the WAITING state:

Object.wait() does not set timeout

Thread.join() does not set timeout

LockSupport.park() does not set timeout

Calling the following method will enter the TIMED_WAITING state:

Object.wait(time)

Thread.join(time)

Thread.sleep(time)

LockSupport.parkNanos(time)

LockSupport.parkUntil(time)

1.6 Related scripts

An example script for collecting thread dumps is as follows:

The above is the solution to the exhaustion of Logic execution threads. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.