Shulou(Shulou.com)06/03 Report--

USE method for Resource Analysis

For each resource, check:

1. Utilization utilization

2. Saturation saturation

3. Errors error

Linux 60 second analysis

This content comes from the performance analysis god and the Netflix performance engineering team.

1 、 uptime

2. Dmesg | tail

3 、 vmstat 1

4. Mpstat-P ALL 1

5 、 pidstat 1

6. Iostat-xz 1

7. Free-m

8. Sar-n DEV 1

9. Sar-n TCP,ETCP 1

10 、 top

Uptime

This is a quick way to view the average load, which indicates the number of tasks (processes) to run. On Linux systems, these numbers include processes to run on CPU and processes that block in uninterruptible I / O (usually disk I / O). This gives a high-level concept of resource load (or requirements), which can then be further explored using other tools.

When you respond to a problem for the first time, you can check the average load to see if the problem still exists. In a fault-tolerant environment, servers that encounter performance problems may be automatically removed from the service when you log in to view. The 15-minute average load is too high, while the 1-minute average load is too low, indicating that you logged in too late to find the problem.

Dmesg | tail

[1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [...] [1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child [1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB,file-rss:0kB [2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. CheckSNMP counters.

This will display the past 10 system messages, if any, looking for errors that may cause performance problems.

The above examples include connection dropping caused by OOM killer and TCP SYN flooding. The TCP message even points to your next analysis area: the SNMP counter.

Vmstat 1

This is a virtual memory statistics tool that originated from BSD, and it also shows other system metrics.

Note that the first line of numbers is the summary since startup (except for memory counters).

Mpstat-P ALL 1

$mpstat-P ALL 1 [...] 03:16:41 AM CPU% usr% nice% sys% iowait% soft% steal% guest% gnice% idle03:16:42 AM all 14.27 0.00 0.75 0.44 0.00 0.00 0.06 0.00 84.4803 AM CPU% usr% nice% sys% irq% steal% guest% gnice% idle03:16:42 AM all 14.27 0.00 0.00 0.00 0. . 00 0.00 0.0003:16:42 AM 10.00 0.00 0.00 0.00 100.0003:16:42 AM 2 8.08 0.00 0.00 0.00 91.9203:16:42 AM 3 10.00 0.00 1.00 0.000 . 00 0.00 1.00 0.00 0.00 88.0003:16:42 AM 4 1.01 0.00 0.00 0.00 98.9903:16:42 AM 5 5.10 0.00 0.00 0.00 94.9003:16:42 AM 6 11.00 0.00 0.00 0.00 89.0003:16:42 AM 7 10.00 0.00 0.00 0.00 90.00 [...]

This command displays the time divided into states by CPU time.

The output shows a problem: CPU 0 reaches 100% user time, which is evidence of a single-threaded bottleneck.

It is also important to note whether there are situations where the iowait is high. And% sys time that can be explored through syscall and kernel tracing as well as CPU analysis.

Pidstat 1

Pidstat (1) shows the CPU usage for each process. Top (1) is a popular tool for this purpose; however, pidstat (1) provides scrolling output by default so that you can see changes over time.

$pidstat 1Linux 4.13.0-19-generic (...) 08 Universe 04Universe 2018 _ x86 CPU 6416)

03:20:47 AM UID PID% usr% system% guest% CPU CPU Command03:20:48 AM 0 1307 0.98 0.00 0.98 8 irqbalance03:20:48 AM 33 12178 4.90 0.00 4.90 4 java03:20:48 AM 33 12569 476.47 24.51 0.00 500.98 0 java03:20: 48 AM 0 130249 0.98 0.98 0.00 1.96 1 pidstat

03:20:48 AM UID PID% usr% system% guest% CPU CPU Command03:20:49 AM 33 12178 4.00 0.00 0.00 4 java03:20:49 AM 33 12569 331.00 21.00 0.00 352.00 0 java03:20:49 AM 0 129906 1.00 0.00 0.00 sshd03:20:49 AM 0 130249 1.00 1.00 0.00 2.00 1 pidstat03:20:49 AM UID PID% usr% system% guest% CPU CPU Command03:20:50 AM 33 12178 4.00 0.00 0.00 4.00 4 java03:20:50 AM 113 12356 1.00 0.00 0.00 11 snmp-pass03:20:50 AM 33 12569 210.00 13.00 0.00 223.00 0 java03:20:50 AM 0 130249 1.00 0.00 0.00 1.00 1 pidstat [...]

This output shows that the number of CPU consumed by the Java process per second varies. Note that these percentages are the sum of all the CPU, so 500% equals 100% of the five CPU.

Iostat-xz 1

This tool displays storage device I / O metrics. The output column of each disk device is represented here by a newline character, so it is difficult to read.

Free-m

The-w parameter is recommended, as shown in more detail

Sar-n DEV 1

The sar tool has multiple patterns for different sets of metrics.

Here, I use it to view network device metrics. Check the interface throughput rxkB/s and txkB/s to see if any limits have been reached.

On the figure, you can see that enp2s0f0's received receives traffic at 985 and 2817kb/s, and sended sends out traffic at 847.7 and 965.78kb/s.

Sar-n TCP,ETCP 1

Now, we use sar (1) to view TCP metrics and TCP errors. Columns to check:

Active/s: the number of locally initiated TCP connections per second (for example, through connect ()), actively initiating external connections (which can be understood as client mode)

Passive/s: the number of TCP connections initiated remotely per second (for example, via accept ()), passively receiving external connection requests (which can be understood as server mode)

Retrans/s: the number of TCP retransmissions per second

Active and passive connection counts are useful for characterizing workloads. Retransmission is a sign of a network or remote host problem.

Top

Top command, which can have the function of uptime. In addition, you can sort and search for specified processes by cpu or memory usage. Overall, the top command has powerful process, cpu, and mem bottleneck analysis capabilities. There's not much introduction here.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.