Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of the troubleshooting ideas of the high occupancy rate of CPU in the Linux system, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.

Preface

As Linux operation and maintenance engineers, in our daily work, we will encounter a situation where the CPU load on the Linux server reaches 100%. If the CPU continues to run high, it will affect the normal operation of the business system and cause enterprise losses.

Many OPS students are often at a loss when they encounter this situation. The following two methods are usually used to quickly locate the problem of CPU overload:

Method one

Step 1: use the

Top command, and then sort by shift+p by CPU

Find the pid of the process that consumes too much CPU

Step 2: use the

Top-H-p [process id]

Find the id of the thread that consumes the most resources in the process

Step 3: use the

Echo 'obase=16; [thread id]' | bc or printf "% x\ n" [thread id]

Convert thread id to hexadecimal (lowercase letters)

Bc is the calculator command of linux

Step 4: execute

Jstack [process id] | grep-A 10 [hexadecimal of thread id] "

View thread status information

Method two

Step 1: use the

Top command, and then sort by shift+p by CPU

Find a process that takes up too much CPU

Step 2: use the

Ps-mp pid-o THREAD,tid,time | sort-rn

Get thread information and find the thread that takes up high CPU

Step 3: use the

Echo 'obase=16; [thread id]' | bc or printf "% x\ n" [thread id]

Convert the required thread ID to hexadecimal format

Step 4: use the

Jstack pid | grep tid-A 30 [hexadecimal of thread id]

Print the stack information for the thread

Case analysis

Scene description

Troubleshooting of JAVA process with High CPU occupancy in production Environment

Solution process

1. According to the top command, it is found that the Java process with a PID of 2633 occupies up to 300% of the CPU, resulting in a failure.

2. After finding the process, how to locate the specific thread or code? first, display the list of threads and sort them according to the threads occupied by CPU:

[root@localhost ~] # ps-mp 2633-o THREAD,tid,time | sort-rn

The display results are as follows:

Found the most time-consuming thread (TID) 3626, taking up CPU time for 12 minutes!

3. Convert the required thread TID to hexadecimal format

[root@localhost ~] # printf "% x\ n" 3626e18

4. Finally, use the jstack command to print out the stack information of this thread under the process:

[root@localhost ~] # jstack 2633 | grep "E18"-A 30

Compared with the solution of the fault, finding the fault is equally important! Most of the monitoring software on the market can observe the server load in real time, such as Zabbix, Nagios, Ali Cloud Monitor (for CVM) and so on. But most of the software requires operation and maintenance students to take the initiative to set rules or testing in order to find problems, how to passively receive alarms?

Professor Wang, a practical OPS software, is recommended. For users whose business is deployed on Aliyun, you only need to bind the read-only AcessKey to be monitored to notify the corresponding team members of the alarm information of resources on the cloud in time.

The way of turning active into passive, on the one hand, reduces the work of operation and maintenance engineers, on the other hand, it also reduces the occurrence of operation and maintenance personnel missing or ignoring alarms.

Thank you for reading this article carefully. I hope the article "sample Analysis of troubleshooting ideas with High CPU occupancy rate in Linux system" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.