Shulou(Shulou.com)06/01 Report--

This article will explain in detail what are the daily commands and tools for checking the performance of Linux server. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Overview

You can get a general idea of the use of system resources in 1 minute by using the following commands and tools

Uptime

Dmesg | tail

Vmstat 1

Mpstat-P ALL 1

Pidstat 1

Iostat-xz 1

Free-m

Sar-n DEV 1

Sar-n TCP,ETCP 1

Top

Lsof

Tcpdump

Netstat

Htop

Iotop

IPTraf

Psacct or Acct

Monit

NetHogs

Iftop

Monitorix

Arpwatch

Suricata

VnStat PHP

Nagios

Nmon

Collectl

Some of these commands require the sysstat package to be installed, and some are provided by the procps package. The output of these commands helps quickly locate performance bottlenecks and check the utilization (utilization), saturation (saturation) and error metrics of all resources (CPU, memory, disk IO, etc.), which is the so-called USE method

Let's introduce these commands and tools one by one. For more parameters and instructions about these commands and tools, please refer to the manual.

1.uptime

$uptime23:51:26 up 21:31, 1 user, load average: 30.02,26.43,19.02

This command allows you to quickly check the load of the machine. In Linux systems, this data represents the number of processes waiting for CPU resources and the number of uninterruptible IO processes blocked (process state is D). These data can give us a macroscopic understanding of the use of system resources.

The output of the command represents the average load of 1 minute, 5 minutes, and 15 minutes, respectively. Through these three data, we can know whether the server load tends to be tight or the area is relieved. If the 1-minute average load is high and the 15-minute average load is low, the server is ordering a high load and you need to further investigate where CPU resources are being consumed. On the other hand, if the 15-minute average load is high and the 1-minute average load is low, it is possible that the time of CPU resource shortage has passed.

From the output in the above example, you can see that the average load for the last 1 minute is very high and much higher than the last 15 minutes, so we need to continue to figure out what processes in the current system are consuming a lot of resources. It can be further investigated through vmstat, mpstat and other commands that will be introduced below.

2.dmesg | tail

$dmesg | tail [1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [...] [1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child [1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB, file-rss:0kB [2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.

This command outputs the last 10 lines of the system log. In the output in the example, you can see a kernel oom kill and a TCP packet loss. These logs can help troubleshoot performance problems. Don't forget this step.

3.vmstat-Virtual memory statistics

The vmstat command is used to display more information such as virtual memory, kernel threads, disks, system processes, Imax O modules, interrupts, CPU active status, and so on.

By default, the Linux system does not have the command vmstat, if you want to use it

You must install a package named sysstat. The common usage of command format is as follows:

$vmstat 1procs-memory- swap---io-----system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st34 00 200889792 73708 591828 00 0 5 6 10 96 1 3 032 00 200889920 73708 591860 00 0592 13284 4282 98 11 032 00 200890112 73708 591860 00 9501 2154 99 1 00 032 00 200889568 73712 591856 00 048 11900 2459 99 00 032 00 200890208 73712 591860 00 15898 4840 98 11 00 ^ C

Vmstat (8) command, each line will output some system core indicators, these indicators can give us a more detailed understanding of the system status. This is followed by parameter 1, which indicates that statistics are output once per second, and the header indicates the meaning of each column, which describes some columns related to performance tuning:

R: the number of processes waiting on the CPU resource. This data is a better representation of the CPU load than the average load, and does not contain processes waiting for IO. If this number is greater than the number of CPU cores of the machine, then the machine's CPU resources are saturated.

Free: the amount of memory available to the system (in kilobytes), which can also cause system performance problems if the remaining memory is insufficient. The free command described below can be used to understand the use of system memory in more detail.

Si, so: the number of writes and reads in the swap. If this data is not 0, the system is already using the swap area (swap), and the machine physical memory is insufficient.

Us, sy, id, wa, st: these all represent CPU time consumption, which represents user time (user), system (kernel) time (sys), idle time (idle), IO wait time (wait), and stolen time (stolen, which is generally consumed by other virtual machines).

These CPU times can quickly tell whether the CPU is busy or not. In general, if the sum of user time and system time is very large, CPU is busy executing instructions. If the IO wait time is long, then the bottleneck of the system may be on disk IO.

As you can see from the output of the sample command, a large amount of CPU time is consumed in the user mode, that is, the user application consumes CPU time. This is not necessarily a performance problem and needs to be analyzed in conjunction with the r queue.

4.mpstat-P ALL 1

$mpstat-P ALL 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07Thirty-four CPU) 07:38:49 PM CPU% usr% nice% iowait% irq% soft% steal% guest% gnice% idle07:38:50 PM all 98.47 0.75 0.00 0.00 0.00 0.780738 : 50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.9907:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 2.0007:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.0007:38:50 PM 3 96.97 0.00 0.00 0.00 3.03 [...]

This command can show the occupancy of each CPU, and if one CPU has a particularly high occupancy rate, it may be caused by a single-threaded application.

5.pidstat 1

$pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 rcuos/007:41:03 PM 14 CPU 07:41:02 PM UID PID% usr% system% guest% CPU CPU Command07:41:03 PM 0 9 0.00 0.94 0.00 1 rcuos/007:41:03 PM 0 4214 5.66 5.66 0 . 00 11.32 15 mesos-slave07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 pidstat07:41:03 PM UID PID% usr% system% guest% CPU CPU Command07:41:04 PM 0 4214 6.00 2.00 8.00 15 mesos-slave07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27 java07:41:04 PM 0 6564 1573.00 10.00 1583.00 28 java07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 snmp-pass07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 pidstat ^ C

The pidstat command outputs the CPU utilization of the process, which continues to output and does not overwrite the previous data, making it easy to observe the dynamics of the system. As shown in the output above, you can see that the two JAVA processes take up nearly 1600% of the CPU time, consuming about 16 CPU cores of computing resources.

6.iostat-input / output statistics

Iostat is a simple tool for collecting and displaying system input and output storage device statistics

This tool is commonly used to find performance problems with storage devices, including devices, local disks, such as NFS remote disks

$iostat-xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 steal 14 avg-cpu:% user% nice% system% iowait% steal% idle 73.96 0.00 3.73 0.03 0.06 22.21Device: rrqm/s wrqm/s rmer s Wamp s rkB/s wkB/s avgrq-sz avgqu-sz await r _ Await w_await svctm% utilxvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09xvdb 0.01 0.00 1.02 8.94 127.97 598.53 0.43 1.78 0.28 0.2500 .25xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.00dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03 [.] ^ C

The iostat command is mainly used to view the machine disk IO. The columns output by this command, the main meanings are:

RComps, wUnip s, rkB/s, wkB/s: represent the number of reads and writes per second and the amount of data read and written per second (kilobytes), respectively. Too much reading and writing can cause performance problems.

The average wait time for await:IO operations, in milliseconds. This is the time it takes for the application to interact with the disk, including the IO wait and the actual operation. If this number is too large, the hardware may encounter a bottleneck or malfunction.

Avgqu-sz: the average number of requests made to the device. If this value is greater than 1, it may be that the hardware device is saturated (some front-end hardware devices support parallel writing).

% util: device utilization. This value indicates how busy the device is, and the empirical value is that if it exceeds 60, it may affect IO performance (you can refer to the average wait time of IO operations). If it reaches 100%, the hardware is saturated.

If the data of the logical device is displayed, the device utilization does not mean that the actual hardware device at the back end is saturated. It is worth noting that even if IO performance is not satisfactory, it does not necessarily mean that the application performance will be poor, and strategies such as pre-read and write caching can be used to improve application performance.

7.free-m

$free-m total used free shared buffers cachedMem: 245998 24545 221453 83 59 541 + buffers/cache: 23944 222053Swap: 0 0

The free command can view the usage of system memory, and the-m parameter indicates that it is displayed in megabytes. The last two columns represent the amount of memory used for IO caching and the amount of memory used for file system page caching, respectively. Note that the second line, / + buffers/cache, looks like the cache takes up a lot of memory space. This is the memory usage strategy of the Linux system, using memory as much as possible, and if the application needs memory, this memory will be immediately reclaimed and allocated to the application. As a result, this part of memory is generally treated as available memory.

If there is very little available memory, the system may use the swap area (if configured), which will increase the IO overhead (which can be withdrawn in the iostat command) and slow system performance.

8.sar-n DEV 1

$sar-n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 _ x86 CPU 64 _ (32) 12:16:48 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s% ifutil12:16:49 AM eth0 18763.00 20686.42 478.30 0.00 0.00 0.0012 : 16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.0012:16:49 AM docker0 0.00 0.00 0.00 0.0012:16:49 AM IFACE rxpck/s txpck/s rxkB/s txkB/s .00 0.00 0.00 0.00 ^ C

The sar command can check the throughput of network devices here. When troubleshooting performance problems, you can determine whether the network device is saturated by the throughput of the network device. For example, in the sample output, the throughput rate of the eth0 network card device is about 22 Mbytes/s, or 176 Mbits/sec, which does not reach the hardware upper limit of 1Gbit/sec.

9.sar-n TCP,ETCP 1

$sar-n TCP ETCP 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07 titanclusters-xxxxx 14 AM atmptf/s estres/s retrans/s isegerr/s orsts/s12:17:20 AM 2015 _ x86 pound 6432 CPU 12:17:19 AM active/s passive/s iseg/s oseg/s12:17:20 AM 1.00 0.00 10233.00 18846.0012 titanclusters-xxxxx 17 purl 19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s12:17:20 AM 0.00 0.00 0.00 .00 0.0012 17 AM active/s passive/s iseg/s oseg/s12:17:21 AM 20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s12:17:21 AM 1.00 0.00 8359.00 6039.0012 15 15 0.00 0.00 0.00 ^ C

The sar command is used here to view the status of the TCP connection, including:

Active/s: the number of locally initiated TCP connections per second, both TCP connections created through connect calls

Passive/s: the number of TCP connections initiated remotely per second, that is, TCP connections created through accept calls

Retrans/s: number of TCP retransmissions per second

The number of TCP connections can be used to determine whether the performance problem is due to the establishment of too many connections, and further to determine whether the connection is initiated actively or passively accepted. TCP retransmission may be due to poor network environment, or excessive pressure on the server resulting in packet loss.

10.top-Linux system process monitoring

The top command is a performance monitor that can be used in many Linux/Unix versions, and it is also a tool often used by Linux system administrators to monitor system performance. The Top command periodically displays all running and actual running and updates to the list, showing CPU usage, memory usage, swap memory, cache size, buffer size, process control, users, and more commands. It also shows running processes with excessive memory and CPU usage. The top command is very useful for system administrators when we need to monitor and take correct action on the Linux system. Let's look at the actual operation of the top command.

Toptop-00:15:40 up 21:56, 1 user, load average: 31.09,29.87, 29.92Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie%Cpu (s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem: 25190241+total, 24921688 used, 22698073+free, 60448 buffersKiB Swap: 0 total, 0 used 0 free. 554208 cached Mem PID USER PR NI VIRT RES SHR S% CPU% MEM TIME+ COMMAND 20248 root 20 0 0.227t 0.012t 18748 S 3090 5.229812 java 4213 root 20 2722544 64640 44232 S 23.5 233R1.00.0 root 00.07 top 5235 root 20 038.227g 54700449996 S 0.70.2 2:02.74 java 4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 java 1 root 20 0 33620 2920 1496 S 0.0 0.0 0:03.82 init 2 root 20 00 00 S 0.0 0.0 0:00.02 kthreadd 3 root 20 00 00 S 0.0 0.00: 05.35 ksoftirqd/0 5 root 0-20 000 S 0.0 0.00: 00.00 kworker/0:0H 6 root 20 00 00 S 0.0 0.00: 06.94 kworker/u256:0 8 root 20 00 S 0.0 2 root 38.05 rcu_sched

The top command contains checks for several previous commands. For example, system load (uptime), system memory usage (free), system CPU usage (vmstat) and so on. So with this command, you can get a relatively comprehensive view of the source of the system load. At the same time, the top command supports sorting, which can be sorted by different columns, making it easy to find processes such as the processes with the most memory, the processes with the highest CPU usage, and so on.

However, the output of the top command is an instantaneous value relative to the previous commands, and if you don't keep staring, you may miss some clues. At this point, you may need to pause the top command refresh to record and compare the data.

11.lsof-Open the file list

The lsof command is available on many Linux/Unix systems, mainly displaying open files and processes in the form of lists.

Open files mainly include disk files, network sockets, pipes, devices and processes. The main reason for using this command is that a disk cannot be unloaded and displays an error message that the file is in use or opened. This command makes it easy to see which files are in use

The most commonly used format for this command:

12.tcpdump-Network packet Analyzer

Tcpdump is the most widely used command line network packet analyzer or packet sniffer, which is mainly used to capture and filter specific excuse information that TCP/IP packets receive or transfer to a network. It also provides an option parameter to save the captured package in a file for later analysis, and tcpdump is available in almost all Linux versions.

13.netstat-Network Statistics

The netstat command is a command-line tool that monitors the statistical interface of incoming and outgoing network packets

It is a very useful tool for many system administrators to monitor network performance and solve network-related problems.

14.htop-process Monitoring

Htop is a more advanced interactive real-time monitoring tool. Htop is very similar to the top command, but it has some very rich features, such as user-friendly interface management processes, keyboard shortcuts, horizontal and vertical processes, and more. Htop is a third-party tool and is not included in the Linux system. You need to install it using the package management tool.

15.iotop-Monitoring Linux disk iBond O

Iotop is also similar to the top and htop commands, but it has a reporting function to monitor and display real-time disk input and output and program processes. This tool is very useful for finding accurate high disk read / write processes

16.IPTraf-Real-time IP LAN Monitoring

IPTraf is a real-time network (IP network) monitoring tool based on open source Linux system. It can collect all kinds of information, such as monitoring IP traffic through the network, including TCP flag information, ICMP details, TCP/UDP traffic failures, TCP connected packets and Byrne count.

And it also collects interface statistics of general and detailed information such as TCP,UDP,ICMP,IP, non-IP,IP parity errors, interface activity, etc.

17.Psacct or Acct-monitors user activity

Psacct or Acct is a very useful tool for monitoring the active status of each user to the system. There are two daemons running in the background, one is closely watching the overall activity of each user on the system, and the other is concerned about what resources are consumed by them.

This tool is very useful for system administrators to track the activities of each user and to know what the user is doing, what kind of commands are issued, how many resources are being used, and how long they are active on the system.

18.Monit-Program and Service Monitoring

This is a free and open source web program based on automatic monitoring and management of system processes, programs, files, directories, permissions, verification file system

The services it monitors include Apache, MYSQL, Mail, FTP, Nginx and so on. The system status can be viewed from the command line or from your own network interface

19.NetHogs-monitors the network bandwidth of each process

NetHogs is a beautiful open source Mini Program (similar to the top command on Linux)

Keep each process network active on your system. It also maintains the real-time network traffic bandwidth usage of a program or application.

20.iftop-Network bandwidth Monitoring

Iftop is another terminal-based open source system monitoring tool whose main function is to display a frequently updated list of network bandwidth utilization (i.e. source and destination hosts) through the network interface on your own system. Iftop monitors network usage, while top monitors CPU usage. Iftop monitors a selected interface and displays the current broadband usage between the two hosts

21.Monitorix-system and network monitoring

Monitorix is a lightweight monitoring tool on Linux/Unix as much as possible, mainly designed to monitor running systems and network resources. It has a built-in HTTP web service to collect system and network information on a regular basis and display it as pictures. It can monitor the average load usage of the system, the allocation of memory, disk drives, system services, network ports, mail statistics (Sendmail, Postfix, Dovecot) MYSQL database and more services. Its main purpose is to monitor the performance of the entire system, and to help monitor failures, bottlenecks, abnormal activities, etc.

22.Arpwatch-Ethernet activity Monitor

Arpwatch is a program used to monitor the address resolution (network address translation) of Ethernet network traffic in Linux networks. It has been monitoring Ethernet traffic and the IP and MAC address pairs that generate logs with the change of network timestamps. When an IP address or MAC address pair changes, it will send an email to notify the administrator; and it is very useful in detecting ARP.

23.Suricata-Network Security Monitoring

Suricata is a high-performance open source network security and * detection and prevention of Linux, FreeBSD, Windows

And other operating system monitoring tools. It is owned by a non-profit fund, OISF (Open Information Security Foundation).

24.VnStat PHP-Monitoring network bandwidth

VnStat PHP is the most popular social tool for web front-end applications called "vnstat". VnStat PHP uses a good graphical mode to monitor the use of network traffic. It shows network traffic usage in hourly, daily, and monthly summary reports.

25.Nagios-Network / Server Monitoring

Nagios is a leading open source and powerful monitoring system where network / system administrators identify and resolve server-related issues before they affect major business processes. Nagios can monitor remote Linux, Windows, switches, single window routers and printers

It can show critical alarms on your network and server, and help you solve problems before mistakes turn back.

26.Nmon-monitors Linux system performanc

The Nmon (Nigel performance Monitor) tool is used to monitor all the resources of the Linux system, including CPU, memory, disk utilization, processes on the network, NFS, kernel, and so on. This tool has two modes: online mode and capture mode. Online mode is suitable for real-time monitoring, and capture mode is used to store processing after output to CSV format.

27.Collectl-Integrated performance testing tool

Collectl is another powerful command-line-based monitoring tool that can be used to collect information about system resources

Including CPU utilization, memory, network, nodes, processes, NFS, TCP sockets, etc.

This is the end of this article on "what are the daily commands and tools for viewing the performance of Linux servers?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.