Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to check the performance of the Linux server, the article is very detailed, has a certain reference value, interested friends must read it!

Overview

You can get a general idea of system resource usage within 1 minute by executing the following command.

Uptime dmesg | tail vmstat 1 mpstat-P ALL 1 pidstat 1 iostat-xz 1 free-m sar-n DEV 1 sar-n TCP,ETCP 1 top

Some of these commands require the sysstat package to be installed, and some are provided by the procps package. The output of these commands helps to quickly locate performance bottlenecks and check the utilization (utilization), saturation, and error metrics of all resources (CPU, memory, disk IO, and so on), which is the so-called USE method.

Let's introduce these commands one by one. For more parameters and instructions on these commands, please refer to the manual of the command.

(1) uptime

$uptime 23:51:26 up 21:31, 1 user, load average: 30.02,26.43,19.02

This command allows you to quickly check the load of the machine. In Linux systems, this data represents the number of processes waiting for CPU resources and the number of uninterruptible IO processes blocked (process state is D). These data can give us a macroscopic understanding of the use of system resources.

The output of the command represents the average load of 1 minute, 5 minutes, and 15 minutes, respectively. Through these three data, we can know whether the server load tends to be tight or the area is relieved. If the 1-minute average load is high and the 15-minute average load is low, the server is ordering a high load and you need to further investigate where CPU resources are being consumed. On the other hand, if the 15-minute average load is high and the 1-minute average load is low, it is possible that the time of CPU resource shortage has passed.

From the output in the above example, you can see that the average load for the last 1 minute is very high and much higher than the last 15 minutes, so we need to continue to figure out what processes in the current system are consuming a lot of resources. It can be further investigated through vmstat, mpstat and other commands that will be introduced below.

(2) dmesg | tail

$dmesg | tail [1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [...] [1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child [1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB, file-rss:0kB [2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.

This command outputs 10 lines of the system log. In the output in the example, you can see a kernel oom kill and a TCP packet loss. These logs can help troubleshoot performance problems. Don't forget this step.

(3) vmstat 1

$vmstat 1 procs-memory--swap---io-----system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 34 00 200889792 73708 591828 00 0 0 5 6 10 96 1 3 00 32 00 200889920 73708 591860 00 0592 13284 4282 98 11 00 32 00 200890112 73708 591860 00 9501 2154 99 1 00 0 32 00 200889568 73712 591856 00 0 48 11900 2459 99 00 00 32 00 200890208 73712 591860 00 15898 4840 98 11 00 ^ C

Vmstat (8) command, each line will output some system core indicators, these indicators can give us a more detailed understanding of the system status. This is followed by parameter 1, which indicates that statistics are output once per second, and the header indicates the meaning of each column, which describes some columns related to performance tuning:

R: the number of processes waiting on the CPU resource. This data is a better representation of the CPU load than the average load, and does not contain processes waiting for IO. If this number is greater than the number of CPU cores of the machine, then the machine's CPU resources are saturated.

Free: the amount of memory available to the system (in kilobytes), which can also cause system performance problems if the remaining memory is insufficient. The free command described below can be used to understand the use of system memory in more detail.

Si, so: the number of writes and reads in the swap. If this data is not 0, the system is already using the swap area (swap), and the machine physical memory is insufficient.

Us, sy, id, wa, st: these all represent CPU time consumption, which represents user time (user), system (kernel) time (sys), idle time (idle), IO wait time (wait), and stolen time (stolen, which is generally consumed by other virtual machines). These CPU times can quickly tell whether the CPU is busy or not. In general, if the sum of user time and system time is very large, CPU is busy executing instructions. If the IO wait time is long, then the bottleneck of the system may be on disk IO.

As you can see from the output of the sample command, a large amount of CPU time is consumed in the user mode, that is, the user application consumes CPU time. This is not necessarily a performance problem and needs to be analyzed in conjunction with the r queue.

(4) mpstat-P ALL 1

$mpstat-P ALL 1 Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 _ x86 March 6432 CPU 07:38:49 PM CPU% usr% sys% iowait% irq% soft% steal% guest% gnice% idle 07:38:50 PM all 98.47 0.00 0.00 0.00 0.78 07:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.99 07:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 2.00 07:38:50 PM 2 98.00 0.00 1.00 0.00 0. 00 0.00 0.00 0.00 1.00 07:38:50 PM 3 96.97 0.00 0.00 0.00 3.03 [...]

This command can show the occupancy of each CPU, and if one CPU has a particularly high occupancy rate, it may be caused by a single-threaded application.

(5) pidstat 1

$pidstat 1 Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 rcuos/0 07:41:02 PM UID PID% usr% system% guest% CPU CPU Command 07:41:03 PM 0 9 0.00 0.94 1 rcuos/0 07:41:03 PM 0 4214 5.66 5. 66 0.00 11.32 15 mesos-slave 07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java 07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java 07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java 07:41:03 PM 60004 60154 0.94 4 .72 0.00 5.66 9 pidstat 07:41:03 PM UID PID% usr% system% guest% CPU CPU Command 07:41:04 PM 0 4214 6.00 2.00 8.00 15 mesos-slave 07:41:04 PM 0 6521 1590.00 1.00 1591.00 27 java 07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28 java 07:41:04 PM 108 6718 1.00 0.00 0.00 snmp-pass 07:41:04 PM 60004 1.00 4.00 0.00 5.00 9 pidstat ^ C

The pidstat command outputs the CPU utilization of the process, which continues to output and does not overwrite the previous data, making it easy to observe the dynamics of the system. As shown in the output above, you can see that the two JAVA processes take up nearly 1600% of the CPU time, consuming about 16 CPU cores of computing resources.

(6) iostat-xz 1

$iostat-xz 1 Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 iowait 14 CPU avg-cpu:% user% nice% system% iowait% steal% idle 73.96 0.00 3.73 0.03 0.06 22.21 Device: rrqm/s wrqm/s Randall s rkB/s wkB/s avgrq-sz avgqu-sz Await r_await w_await svctm% util xvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09 xvdb 1.02 8.94 127.97 598.53 0.00 0.43 1.78 0.28 0.25 0.25 xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26 dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04 dm-1 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00 dm-2 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03 [.] ^ C

The iostat command is mainly used to view the machine disk IO. The columns output by this command, the main meanings are:

RComps, wUnip s, rkB/s, wkB/s: represent the number of reads and writes per second and the amount of data read and written per second (kilobytes), respectively. Too much reading and writing can cause performance problems.

The average wait time for await:IO operations, in milliseconds. This is the time it takes for the application to interact with the disk, including the IO wait and the actual operation. If this number is too large, the hardware may encounter a bottleneck or malfunction.

Avgqu-sz: the average number of requests made to the device. If this value is greater than 1, it may be that the hardware device is saturated (some front-end hardware devices support parallel writing).

% util: device utilization. This value indicates how busy the device is, and the empirical value is that if it exceeds 60, it may affect IO performance (you can refer to the average wait time of IO operations). If it reaches 100%, the hardware is saturated. If the data of the logical device is displayed, the device utilization does not mean that the actual hardware device at the back end is saturated. It is worth noting that even if IO performance is not satisfactory, it does not necessarily mean that the application performance will be poor, and strategies such as pre-read and write caching can be used to improve application performance.

(7) free-m

$free-m total used free shared buffers cached Mem: 245998 24545 221453 83 59 541-/ + buffers/cache: 23944 222053 Swap: 0 0

The free command can view the usage of system memory, and the-m parameter indicates that it is displayed in megabytes. The two columns represent the amount of memory used for IO caching and the amount of memory used for file system page caching, respectively. Note that the second line, / + buffers/cache, looks like the cache takes up a lot of memory space. This is the memory usage strategy of the Linux system, using memory as much as possible, and if the application needs memory, this memory will be immediately reclaimed and allocated to the application. As a result, this part of memory is generally treated as available memory.

If there is very little available memory, the system may use the swap area (if configured), which will increase the IO overhead (which can be withdrawn in the iostat command) and slow system performance.

(8) sar-n DEV 1

$sar-n DEV 1 Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 * . 00 12:16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 12:16:49 AM docker0 0.00 0.00 0.00 12:16:49 AM IFACE rxpck/s txpck/s rxkB/ S txkB/s rxcmp/s txcmp/s rxmcst/s% ifutil 12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 12:16:50 AM lo 20.00 3.25 3.25 0.00 0.00 0.00 12:16:50 AM docker0 0.00 0.00 0.00 ^ C

The sar command can check the throughput of network devices here. When troubleshooting performance problems, you can determine whether the network device is saturated by the throughput of the network device. For example, in the sample output, the throughput rate of the eth0 network card device is about 22 Mbytes/s, or 176 Mbits/sec, which does not reach the hardware upper limit of 1Gbit/sec.

(9) sar-n TCP,ETCP 1

$sar-n TCP ETCP 1 Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 AM atmptf/s estres/s retrans/s isegerr/s orsts/s 14 AM atmptf/s estres/s retrans/s isegerr/s orsts/s 2015 _ x86 August 6432 CPU 12:17:19 AM active/s passive/s iseg/s oseg/s 12:17:20 AM 1.00 18846.00 12:17:19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s 12:17:20 AM 0.00 0.00. 00 0.00 0.00 12:17:20 AM active/s passive/s iseg/s oseg/s 12:17:21 AM 1.00 8359.00 6039.00 12:17:20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s 12:17:21 AM 0.00 0.00 0.00 ^ C

The sar command is used here to view the status of the TCP connection, including:

Active/s: the number of locally initiated TCP connections per second, both TCP connections created through connect calls

Passive/s: the number of TCP connections initiated remotely per second, that is, TCP connections created through accept calls

Retrans/s: number of TCP retransmissions per second

The number of TCP connections can be used to determine whether the performance problem is due to the establishment of too many connections, and further to determine whether the connection is initiated actively or passively accepted. TCP retransmission may be due to poor network environment, or excessive pressure on the server resulting in packet loss.

(10) top

Top top-00:15:40 up 21:56, 1 user, load average: 31.09,29.87,29.92 Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie% Cpu (s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 25190241+total, 24921688 used, 22698073+free, 60448 buffers KiB Swap: 0 total 0 used, 0 free. 554208 cached Mem PID USER PR NI VIRT RES SHR S% CPU% MEM TIME+ COMMAND 20248 root 20 0 0.227t 0.012t 18748 S 3090 5.229812 java 4213 root 20 2722544 64640 44232 S 23.5 233LV 35.37 mesos-slave 66128 titancl+ 20 024344 2332 1172 R 1.0 0.07 top 5235 root 20 0 38.227g 54700449996 S 0.7 0 . 2 2:02.74 java 4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 java 1 root 20 0 33620 2920 1496 S 0.0 0.0 0:03.82 init 2 root 20 00 00 S 0.0 0.0 0:00.02 kthreadd 3 root 20 00 0 0 S 0.0 0.00: 05.35 ksoftirqd/0 5 root 0-20 000 S 0.0 0.00: 00.00 kworker/0:0H 6 root 20 00 00 S 0.0 0.00: 06.94 kworker/u256:0 8 root 20 00 00 S 0.02 kworker/u256:0 38.05 rcu_sched

The top command contains checks for several previous commands. For example, system load (uptime), system memory usage (free), system CPU usage (vmstat) and so on. So with this command, you can get a relatively comprehensive view of the source of the system load. At the same time, the top command supports sorting, which can be sorted by different columns, making it easy to find processes such as the processes that take up the most memory and the processes with the CPU occupancy rate.

However, the output of the top command is an instantaneous value relative to the previous commands, and if you don't keep staring, you may miss some clues. At this point, you may need to pause the top command refresh to record and compare the data.

These are all the contents of the article "how to check the performance of Linux Server". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.