Shulou(Shulou.com)06/01 Report--

What are the Linux performance analysis tools, I believe many inexperienced people are at a loss about this, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

This article is organized out of interest in the Linux operating system and a strong desire for low-level knowledge. This paper can also be used as an index to test the basic knowledge, in addition, the article covers all aspects of a system. If there is no perfect computer system knowledge, network knowledge and operating system knowledge, it is impossible to fully master the tools in the document. in addition, the system performance analysis and optimization is a long-term series.

Performance analysis tool

First, let's look at a picture:

The figure above is a share of a performance analysis of Brendan Gregg, and all the tools in it can get its help documentation through man. The following question briefly introduces the general usage:

Vmstat-- virtual memory statistics

Vmstat (VirtualMeomoryStatistics, Virtual memory Statistics) is a common tool for monitoring memory in Linux, which can monitor the overall situation of virtual memory, process, CPU and so on.

The general usage of vmstat: vmstat interval times is sampled every interval seconds, sampling times times in total. If times is omitted, the data will be collected until the user stops manually. Take a simple example:

You can use ctrl+c to stop vmstat from collecting data.

The first line shows the average of the system since it was started, the second line starts to show what is happening now, and the next line shows what happened at every five-second interval, with the meaning of each column in the header, as follows:

The procs:r column shows how many processes are waiting for the cpu,b column to show how many processes are in uninterruptible sleep (waiting for IO).

The memory:swapd column shows how many blocks have been swapped out of disk (page swapping), and the remaining columns show how many blocks are free (unused), how many blocks are being used as buffers, and how many blocks are being used as operating system cache.

Swap: shows the swap activity: how many blocks are being swapped in (from disk) and out (to disk) per second.

Io: shows how many blocks are read from the block device (bi) and written out (bo), which usually reflects the hard disk IUnio.

System: displays the number of interrupts per second (in) and context switching (cs).

Cpu: shows the percentage of all cpu time spent on various operations, including executing user code (non-kernel), executing system code (kernel), being idle, and waiting for IO.

Performance of insufficient memory: free memory decreased sharply, recovery of buffer and cacher did not help, heavy use of swap partitions (swpd), frequent page swapping (swap), increased number of read and write disks (io), increased page break (in), increased number of context switching (cs), increased number of processes waiting for IO (b), a large amount of CPU time was spent waiting for IO (wa)

Iostat-- is used to report CPU statistics

Iostat is used to report central processing unit (CPU) statistics and input / output statistics for the entire system, adapters, tty devices, disks, and CD-ROM, showing the same cpu usage information as vmstat by default, and using the following command to display extended device statistics:

The first line shows the average since the system was started, and then the average of the increments, one line per device.

Common abbreviations for linux disk IO metrics: rq is request,r is read,w is write,qu is queue,sz is size,an is verage,tm is time,svc is service.

Rrqm/s and wrqm/s: read and write requests merged per second. "merged" means that the operating system takes multiple logical requests from the queue and merges them into a single request to the actual disk.

Ripple s and wUnip s: the number of read and write requests sent to the device per second.

Rsec/s and wsec/s: the number of sectors read and written per second.

Avgrq-sz: the number of sectors requested.

Avgqu-sz: the number of requests waiting in the device queue.

Await: the time spent per IO request.

Svctm: actual request (service) time.

% util: the percentage of time spent with at least one active request.

Dstat-- system monitoring tool

Dstat shows the use of cpu, disk io, network packets and page change. The output is color, readable, and more detailed and intuitive than the input of vmstat and iostat. When in use, you can enter the command directly, of course, you can also use specific parameters.

As follows: dstat-cdlmnpsy

Real-time monitoring tool for iotop--LINUX process

The iotop command is a command that specifically displays the hard disk IO, and the interface style is similar to the top command, which shows which process is responsible for the IO load. Is a top-like tool used to monitor disk IWeiO usage, with UI similar to top, including PID, user, IWeiO, process, and other related information.

Can be used in a non-interactive manner:

Iotop-bod interval, check the I instatpidstat-- O of each process, and you can use pidstat,pidstat-d instatpidstat-- to monitor the system resources.

Pidstat is mainly used to monitor the situation that all or specified processes occupy system resources, such as CPU, memory, device IO, task switching, threads and so on.

How to use it:

Pidstat-d interval # Statistics CPU usage Information pidstat-u interval # Statistics memory Information Pidstat-r intervaltop

The summary area of the top command shows five aspects of system performance information:

Load: time, number of logged-in users, average system load

Process: run, sleep, stop, zombie

Cpu: user mode, kernel mentality, NICE, idle, waiting for IO, interrupt, etc.

Memory: total, used, idle (system perspective), buffer, cache

Swap Partition: total, used, idle

The task area displays by default: process ID, valid user, process priority, nice value, virtual memory used by the process, physical memory and shared memory, process status, CPU utilization, memory occupancy, cumulative CPU time, process command line information.

Htop

Htop is an interactive process viewer in Linux systems, a text-mode application (in the console or X terminal) that requires ncurses.

Htop allows users to operate interactively, support color themes, scroll through the list of processes horizontally or vertically, and support mouse operation.

Compared with top, htop has the following advantages:

You can scroll through the list of processes horizontally or vertically to see all the processes and the complete command line.

Faster than top on startup.

You do not need to enter a process number when killing a process.

Htop supports mouse operation.

Mpstat

Mpstat, the abbreviation of Multiprocessor Statistics, is a real-time system monitoring tool. It reports along with some statistics for CPU, which are stored in the / proc/stat file. In a multi-CPUs system, it can view not only the average status information of all CPU, but also the information of a specific CPU. Common usage:

Mpstat-P ALL interval timesnetstat

Netstat is used to display statistics related to IP, TCP, UDP and ICMP protocols, and is generally used to verify the network connectivity of each port of the machine.

Common usage:

Netstat-npl # can see if the port you want to open is already open. Netstat-rn # prints routing table information. Netstat-in # provides interface information on the system, prints the MTU of each interface, the number of input packets, input errors, output packets, output errors, collisions, and the current length of the output queue. Ps-- displays the status of the current process

There are too many ps parameters. For more information on how to use them, please see man ps.

Common methods:

Ps aux # hsserver ps-ef | grep # hundsun # kill the method ps aux of a program | grep mysqld | grep-v grep | awk'{print $2} 'xargs kill-9 # kill the zombie process ps-eal | awk' {if ($2 = = "Z") {print $4}'| xargs kill-9strace

Track the system calls and received signals generated during the execution of the program to help analyze the abnormal conditions encountered in the execution of the program or command.

For example: to see which configuration file mysqld loads on linux, you can run the following command:

Strace-e stat64 mysqld-print-defaults > / dev/nulluptime

To be able to print how long the system has been running and the average load of the system, the last three numbers output by the uptime command mean the average load of the system within 1 minute, 5 minutes, and 15 minutes, respectively.

Lsof

Lsof (list open files) is a tool that lists files currently open by the system. Through the lsof tool, you can view this list of system detection and troubleshooting, common uses:

# View file system blocking lsof / boot # see which process the port number is occupied by lsof-I: 3306 # see which files the user opens lsof-u username # see which files are opened by the process lsof-p 4838 # check the remote open network link lsof-I @ 192.168.34.128perf

Perf is a system performance optimization tool that comes with Linux kernel. The advantage lies in its close integration with Linux Kernel, which can be first applied to new feature that joins Kernel to view hotspot functions and the ratio of cashe miss, thus helping developers to optimize program performance.

The basic principle of performance tuning tools such as perf,Oprofile is to sample the monitored object. The simplest case is to sample according to the tick interrupt, that is, the sampling point is triggered within the tick interrupt, and the context of the program is judged in the sampling point. If a program spends 90% of its time on the function foo (), then 90% of the sampling points should fall in the context of the function foo (). Luck is elusive, but I think the above inference is more reliable as long as the sampling frequency is high enough and the sampling time is long enough. Therefore, through tick trigger sampling, we can understand which parts of the program are the most time-consuming, thus focusing on the analysis.

Summary: combined with the above commonly used performance test commands and the diagrams of the performance analysis tools at the beginning of the article, you can have a preliminary understanding of which aspects of performance in the process of performance analysis use which aspects of tools (commands).

Commonly used performance testing tools

Proficient and proficient in the second part of the performance analysis command tools, the introduction of several performance testing tools, before the introduction of a few performance testing tools:

Perf_events

A performance diagnostics tool released and maintained with Linux kernel code, maintained and developed by the kernel community. Perf can be used not only for application performance statistics and analysis, but also for kernel code performance statistics and analysis.

EBPF tools

A performance tracking tool using bcc, eBPF map can use custom eBPF programs that are widely used in kernel tuning and can also read user-level asynchronous code. The important thing is that this external data can be managed in user space. The map data body in KMurv format is managed by calling bpf system calls in user space to create, add, delete, and so on.

Perf-tools:

A Linux performance analysis and tuning tool set based on perf_events (perf) and ftrace. Perf-Tools depends on few libraries and is easy to use. Linux 3.2 and above kernel versions are supported.

Bcc (BPF Compiler Collection)

A perf performance analysis tool using eBPF. A toolkit for creating efficient kernel tracing and operating programs, including several useful tools and examples. With the extended BPF (Berkeley packet filter), officially known as eBPF, a new feature was first added to Linux 3.15. BCC above Linux 4.1 is required for multi-purpose.

Ktap

A new dynamic performance tracking tool for linux scripts. Allows users to track Linux kernel dynamics. Ktap is designed for interoperability, allowing users to adjust operations, troubleshooting and extending kernels and applications. It is similar to Linux and Solaris DTrace SystemTap.

Flame Graphs

Is a visual graphics software using perf,system tap,ktap, allowing the most frequent code paths to be quickly and accurately identified, which can be generated using the development source code in github.com/brendangregg/flamegraph.

Linux observability tools | Linux performance observation tool

The first Basic Tool to learn is as follows:

Uptime, top (htop), mpstat, isstat, vmstat, free, ping, nicstat, dstat.

The advanced commands are as follows:

Sar 、 netstat 、 pidstat 、 strace 、 tcpdump 、 blktrace 、 iotop 、 slabtop 、 sysctl 、 / proc .

Linux benchmarking tools | Linux performance evaluation tool

Is a performance evaluation tool, for the performance testing of different modules can use the corresponding tools, for in-depth understanding, you can refer to the attachment document below.

Linux tuning tools | Linux performance tuning tool

Is a performance tuning tool, mainly from the linux kernel source layer tuning, for in-depth understanding, you can refer to the attached document below.

Linux observability sar | linux performance observation tool

Sar (System Activity Reporter system activity report) is one of the most comprehensive system performance analysis tools on LINUX, which can report system activities from many aspects, including: reading and writing of files, usage of system calls, disk ID O, CPU efficiency, memory usage, process activities and IPC-related activities.

The common usage of sar:

Sar [options] [- A] [- o file] t [n]

Where:

T # is the sampling interval, n is the number of samples, and the default value is 1;-o file # means to store the command results in a file in binary format, and file is the file name. Options # as a command line option, have you mastered the methods of the Linux performance analysis tool after reading the above? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.