Shulou(Shulou.com)06/02 Report--

How to understand the average load of Linux? I believe many inexperienced people are at a loss about it. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

The average load can be familiar and unfamiliar to us, but we ask what the average load is, but most people reply that isn't the average load the CPU usage per unit time? In fact, this is not the case, if you can, you can man uptime to learn more about the average load.

What is the average load?

The average load can be familiar and unfamiliar to us, but we ask what the average load is, but most people reply that isn't the average load the CPU usage per unit time? In fact, this is not the case, if you can, you can man uptime to learn more about the average load.

To put it simply, the average load refers to the average number of runnable and uninterruptible processes per unit time, that is, the average number of active processes, which is not directly related to CPU utilization. Explain the words runnable and uninterruptible here.

Runnable status:

Refers to a process that is using CPU or waiting for CPU. We use the ps command to view the process in R state.

Uninterruptible state:

Processes are processes that are in kernel state critical processes, and these processes are uninterruptible. For example, it is common to wait for a response from the hardware device iAccord O, that is, we look at the process in state D in the ps command.

For example, when a process reads and writes data to the disk, in order to ensure the consistency of the data, it cannot be interrupted or interrupted by other processes before it gets a reply from the disk. At this time, the process is in an uninterruptible state. If the process is interrupted at this time, it is prone to the problem of inconsistency between disk data and process data.

Therefore, the uninterruptible state is actually a protection mechanism for system processes and hardware devices.

Therefore, you can simply understand that the average load is the average number of active processes. The average number of active processes is intuitively understood as the number of active processes per unit time, but it is actually the exponential decay average of the number of active processes. Since it is the average number of active processes, ideally there is just one process running on each CPU, so that each CPU will be fully utilized. For example, what does it mean when the average load is 2?

On a system with only 2 CPU, it means that all CPU happens to be fully occupied

On 4 CPU systems, that means CPU is 50% idle

On a system with only one CPU, it means that half of the processes are less than CPU

Average load and CPU utilization

In real life, we often confuse average load with CPU usage, so I'll do a partition here.

You may wonder, since the average load represents the number of active processes, doesn't the high average load mean high CPU usage?

Let's go back to the meaning of average load, which refers to the number of processes that are runnable and uninterruptible per unit of time, so it includes not only processes that normally use CPU, but also processes waiting for CPU and waiting for CPU O.

CPU utilization is the statistics of the busy situation of CPU per unit time, which does not necessarily correspond to the average load, for example:

CPU-intensive processes, the use of a large amount of CPU will lead to an increase in average load, when the two are consistent

CPU O-intensive process, waiting for it will also lead to an increase in average load, but the usage rate of IMAX is not necessarily very high.

The scheduling of a large number of processes waiting for CPU will also lead to an increase in average load, and the utilization rate of CPU will be very high.

Average load case

Here we need to install several tools sysstat, stress, stress-ng

Here the sysstat version of Centos will be a little older, so it's best to upgrade to the latest version. Manual rpm installation or source code installation

Scenario 1. CPU-intensive

1. Run a stress command to simulate a scenario with 100% CPU utilization

Stress-- cpu 1-- timeout 600

2. Open the second terminal and uptime to check the changes of the average load.

$watch-d uptime

09:40:35 up 80 days, 18:41, 2 users, load average: 1.62, 1.10, 0.87

3. Open the third terminal and mpstat to check the changes in CPU utilization.

Mpstat-P ALL 5 20 10:06:37 AM CPU% usr% nice% sys% iowait% irq% steal% guest% gnice% idle 10:06:42 AM all 31.50 0.00 0.35 0.00 0.00 0.00 68.15 10:06:42 AM 0 1.20 0.00 0.80 0.00 0.00 0.00 98.00 10:06:42 AM 1.7.21 0. 00 0.40 0.00 0.00 0.00 92.38 10:06:42 AM 2 100.00 0.00 0.00 0.00 10:06:42 AM 3 17.43 0.00 0.20 0.00 0.00 0.00 82.36 #-P ALL means monitoring all CPU The following number 5 indicates that the data is output at an interval of 5 seconds

From the second terminal, we can see that the average load per minute increases to 1.62. from the third terminal, we can see that there is a CPU utilization rate of 100%, but the iowait is 0, indicating that the increase in the average load is officially caused by the CPU utilization rate of 100%.

So which process caused the CPU usage to be 100%? We can use pidstat to see:

# output data every 5 seconds

Pidstat-u 5 1 10:08:41 AM UID PID% usr% system% guest% wait% CPU CPU Command 10:08:46 AM 0 1 0.20 0.00 0.00 0.00 0.20 systemd 10:08:46 AM 0599 0.00 1.00 0.20 1.00 systemd-journal 10:08:46 AM 0 1043 0.60 0.00 0.00 0.600 rsyslogd 10:08:46 AM 0 6863 100.00 0.00 0.00 100.00 3 stress 10:08:46 AM 0 7303 0.20 0.20 0.00 0.00 0.40 2 pidstat

From here we can see that it is caused by the process of stress.

Scenario 2. Imax O-intensive processes

1. We use the stress-ng command, but this time we simulate the Imax O pressure and keep executing sync:

#-- hdd means to read and write temporary files #-I generate several worker loops to call sync () to generate io pressure $stress-ng-I 4-- hdd 1-- timeout 600

2. Open the second terminal and run uptime to check the average load.

$watch-d uptime 10:30:57 up 98 days, 19:39, 3 users, load average: 1.71,0.75,0.69

3. Open the third terminal and run mpstat to check the CPU utilization

Mpstat-P ALL 5 20 10:32:09 AM CPU% usr% nice% sys% iowait% soft% steal% guest% gnice% idle 10:32:14 AM all 6.80 0.00 33.75 26.16 0.00 0.39 0.00 0.00 32.90 10:32:14 AM 0 4.03 0.00 69.57 19.91 0.00 0.00 0.00 6.49 10:32:14 AM 1 25.32 0. 00 9.49 0.00 0.00 0.95 0.00 0.00 0.00 64.24 10:32:14 AM 2 0.24 0.00 10.87 63.04 0.00 0.48 0.00 0.00 0.00 25.36 10:32:14 AM 3 1.42 0.00 36.93 14.20 0.00 0.28 0.00 0.00 0.00 47.16

As you can see here, the average load per minute will slowly increase to 1.71, and the CPU utilization of one of the CPU's systems will rise to 63.04. This shows that the increase in average load is due to the increase in iowait.

So which process caused us? We use pidstat to see:

$pidstat-u 51 Average: UID PID% usr% system% guest% wait% CPU CPU Command Average: 0 1 0.00 0.19 0.00 0.00 0.19-systemd Average: 0 10 0.00 0.19 0.00 1.56 0.19-rcu_sched Average: 0 599 0.58 1.75 0.00 0.39 2.33-systemd-journal Average: 0 1043 0.19 0.19 0.00 0.00 0.39-rsyslogd Average: 0 6934 0.00 1.56 0.00 1.17 1.56-kworker/2:0-events_power_efficient Average: 0 7383 0.00 0.39 0.00 0.78 0.39-kworker/1:0-events_power_efficient Average: 0 9411 0.00 0.19 0.58 0.19-kworker/0:0-events Average: 0 9662 0.00 97.67 0.00 0.19 97.67-kworker/u8:0+flush-253:0 Average: 0 10793 0.00 0.97 0. 00 1.56 0.97-kworker/3:2-mm_percpu_wq Average: 0 11062 0.00 21.79 0.00 0.19 21.79-stress-ng-hdd Average: 0 11063 0.00 1.95 0.00 1.36 1.95-stress-ng-io Average: 0 11064 0.00 2.72 0.00 0.39 2.72-stress-ng-io Average: 0 11065 0.00 1.36 0.00 1.75 1.36-stress-ng-io Average : 0 11066 0.00 2.72 0.00 0.58 2.72-stress-ng-io

It can be found that stress-ng caused it.

Scenario 3. The scene of a large number of processes

When the running process in the system exceeds the running capacity of CPU, there will be a process waiting for CPU.

For example, we use stress, but this time we simulate eight processes:

$stress-c 8-timeout 600

Our system has only 4 CPU. It is obviously not enough to run 8 processes at this time. The system is seriously overloaded after CPU, and the load value reaches more than 4 points at this time:

$uptime 10:56:22 up 98 days, 20:05, 3 users, load average: 4.52,2.82,2.67

Then let's run pidstat to take a look at the process:

$pidstat-u 5 1 Linux 5.0.5-1.el7.elrepo.x86_64 (k8s-m1) 07Action11Universe 2019 _ x86pixel 644 CPU 10:57:33 AM UID PID% usr% guest% wait% CPU CPU Command 10:57:38 AM 01 0.20 0.00 0.00 0.20 1 systemd 10:57:38 AM 0599 0.00 0.00 0.20 0.99 2 systemd-journal 10:57:38 AM 0 1043 0.60 0.20 0.00 0.00 0.79 1 rsyslogd 10:57:38 AM 0 12927 51.59 0.00 0.00 48.21 51.59 0 stress 10:57:38 AM 0 12928 44.64 0.00 0.00 54.96 44.64 0 stress 10:57:38 AM 0 12929 45.44 0.00 0.00 54.56 45.44 2 stress 10:57:38 AM 0 12930 45.44 0.00 0.00 54.37 45.44 2 stress 10:57:38 AM 0 12931 51.59 0.00 0.00 48.21 51.59 3 stress 10:57:38 AM 0 12932 48.41 0.00 0.00 51.19 48.41 1 stress 10:57:38 AM 0 12933 45.24 0.00 0.00 54.37 45.24 3 stress 10:57:38 AM 0 12934 48.81 0.00 0.00 50.99 48.81 1 stress 10:57:38 AM 0 13083 0.00 0.40 0.00 0.20 0.40 0 pidstat

It can be seen that 8 processes preempted 4 CPU, and each process waited for 50% of the CPU time (% wait), which exceeded the CPU computing power of the process, resulting in CPU overload.

After reading the above, have you mastered how to understand the average load of Linux? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.