Shulou(Shulou.com)06/03 Report--

Check the number of system cpu:

(1) grep 'model name' / proc/cpuinfo | wc-l

(2) grep 'core id' / proc/cpuinfo | wc-l

Installation of performance tools:

(1) centos:yum install-y stress sysstat

(2) ubuntu:apt install stress sysstat

Actual combat

Scenario 1: CPU-intensive processes

(1) pressure test cpu:stress-- cpu 1-- timeout

(2) check the changes in the average load of the system: watch-d uptime (- d indicates the area where the change is highlighted)

(3) check the changes in the cpu utilization of the system: mpstat-P ALL 5 (- P ALL means to monitor all cpu, followed by the number 5 means to output a set of data at intervals of 5 seconds) dynamic output

(4) check the use of cpu by the system process: pidstat-u 52 (- u means to output a set of data every 5 seconds, a total of two times, and print the average value)

Scenario 2: Imax O-intensive processes

Pressure test IO:stress-- io 1-- timeout 600

The process is like scenario one

Scenario 3: a scene of a large number of processes

Simulate 8 processes: stress-- cpu 8-- timeout 600

The process is like scenario one

Scenario 4: view the system context switch

View context switch: vmstat 5 (output 1 set of data every 5 seconds)

Linux performance optimization reading notes 01

(1) cs (context switch) is the number of context switches per second.

(2) number of interrupts per second of in (interrupt)

(3) r (running or runnable) indicates the length of the ready queue, that is, the number of processes running and waiting for CPU

(4) b (blocked) indicates the number of processes in an uninterruptible sleep state

View detailed context switching: pidstat-w 5 (output 1 set of data every 5 seconds, dynamic)

Linux performance optimization reading notes 01

(1) cswch indicates the number of voluntary context switches (voluntary context switches) per second

(2) nvcswch indicates the number of involuntary context switches (non voluntary context switches) per second

Context switching concept:

(1) Voluntary context switching: refers to the context switch caused by the process's inability to obtain the required resources. For example, when system resources such as Icano and memory are insufficient

(2) involuntary context switching: a context switch in which a process is forced to be scheduled by the system because the time slice has expired. For example, when a large number of processes are competing for CPU

Simulate multithreaded scheduling:

(1) centos:yum install-y sysbench

(2) ubuntu:apt install sysbench

Test:

(1) run the benchmark test with 10 threads for 5 minutes to simulate the problem of multi-thread switching in the system: sysbench-- threads=10-- max-time=300 threads run

(2) observe the context switch: vmstat 1

Linux performance optimization reading notes 01

(3) check the processes and threads of cpu and context switching: pidstat-w-u 1

Linux performance optimization reading notes 01

The ready queue of the system is too long, that is, there are too many processes running and waiting for cpu, which leads to a large number of context switching, which in turn leads to the increase of the occupancy rate of system cpu.

(4) the pidstat in the figure above displays the metric data of the process by default, and the metric of the thread will not be output until the-t parameter is added.

Linux performance optimization reading notes 01

(5) find the source and pay attention to the number of interrupts. What type of interruption is rising: watch-d cat / proc/interrupts

You will find that it is RES (rescheduling interrupt), which means that the idle CPU is awakened to schedule new tasks to run.

Scenario 5: view CPU usage

Use the tools top and ps:

(1) top: displays the overall CPU and memory usage of the system, as well as the resource usage of each process. It is refreshed every 3 seconds by default.

Linux performance optimization reading notes 01

The blank line is followed by real-time information for the process, and each process has a% CPU column that indicates the CPU usage of the process.

It is the sum of user-mode and kernel-mode CPU usage, including the CPU used by the process user space, the kernel space CPU executed through system calls, and the CPU waiting to run in the ready queue. In a virtualized environment, it also includes the CPU used to run virtual machines.

Top does not subdivide the user mode CPU and kernel mode CPU of the process, so how to view the details of each process: pidstat 1 5 (outputs a set of data every 1 second, a total of 5 sets)

(2) ps: only show the resource usage of each process

CPU high troubleshooting tool: want to know which function in the code is occupying CPU, find it, you can optimize it more efficiently and specifically.

(1) GDB (the GNU Project Debugger) is a powerful program debugging tool, but it is not suitable for the early application of performance analysis, because the process of GDB debugging program will interrupt the program running, which is often not allowed in the online environment.

(2) perf (a performance analysis tool built in after Linux2.6.31), which is based on performance event sampling, can not only analyze various events and kernel performance of the system, but also be used to analyze the performance problems of specified applications. If the system is not available, please install: yum install-y perf

Two common uses of perf:

(1) perf top is similar to top in that it can display the functions or instructions that take up the most CPU clock in real time, so it can be used to find hot functions, as follows:

Linux performance optimization reading notes 01

The first row contains three data, namely: number of samples (Samples), event type (event) and total number of events (Event count)

Also note: if the number of samples is too small (for example, there are more than a dozen), then the following sort and percentage will have no practical reference value.

Further down is a tabular style of data, each row containing four columns, which are:

The first column, Overhead, is the proportion of the symbol's performance events in all samples, expressed as a percentage.

The second column, Shared, is the dynamic shared object (Dynamic Shared Object) where the function or instruction resides, such as kernel, process name, dynamic link library name, kernel module name, and so on.

The third column, Object, is the type of dynamically shared object. For example, [.] Represents an executable program in user space, or a dynamic link library, while [k] represents kernel space.

The fourth column Symbol is the symbolic name, that is, the function name. When the function name is unknown, it is represented by a hexadecimal address.

(2) perf record and perf report

Although perf top shows the performance information of the system in real time, its disadvantage is that it can not save data, that is, it can not be used for offline or subsequent analysis.

Perf record provides the function of saving data, and the saved data needs to be parsed and displayed with perf report.

In practice, we often add-g parameters to perf top and perf record to enable the sampling of call relationships, which makes it convenient for us to analyze performance problems according to the call chain.

For example: perf top-g-p 21515 (- g enables call relationship analysis, and-p specifies that the process number of the service is 21515)

HTTP service performance testing tool: ab (apache bench)

For example: ab-c 10-n 1000 http://192.168.246.191:80/ (10 concurrent requests for a total of 1000 requests tested)

Scenario 6: the CPU utilization of the system is very high, but no application with high CPU can be found.

Pidstat-p 24344 (specify a process with a PID of 24344)

Pstree displays the relationships between all processes in a tree

Grep stress-r app/ (- r directory recursion)

Execsnoop: a tool specially designed for segmented processes. It monitors the exec () behavior of the process in real time through ftrace, and outputs the basic information of the short-term process, including the PID of the process, the PID of the parent process, the command line parameters and the results of the execution.

CPU usage, which cannot be explained by conventional methods, may be the following two situations:

(1) other binary programs are directly called in the application, which usually take a short time to run and are not easy to find through tools such as top.

(2) the application itself is constantly collapsing and restarting, and the resource initialization of the startup process is likely to take up a lot of CPU.

For such processes, we can use pstree or execsnoop to find their parent process, and then start with the application of the parent process to troubleshoot the root cause of the problem.

Pay attention to ⚠️: when you encounter an unexplained CPU usage problem, first check to see if the short-term application is causing trouble!

Scenario 7: what if there are a large number of uninterruptible processes and zombie processes in the system (part 1)

Linux performance optimization reading notes 01

Five common states of a process:

(1) R is an acronym for Running or Runnable, indicating that a process is running or waiting to run in the CPU's ready queue.

(2) D is the abbreviation of Disk Sleep, that is, uninterruptible state sleep (Uninterruptible Sleep), which generally means that the process is interacting with the hardware, and the interaction process is not allowed to be interrupted by other processes or interruptions.

Note ⚠️: the process is in an uninterruptible state for a long time, which usually indicates that the system has a performance problem.

(3) Z is the abbreviation of Zombie. If you have ever played "plants vs. Zombies", you should know what it means. It means that the zombie process, that is, the process has actually ended, but the parent process has not reclaimed its resources (such as the process descriptor, PID, etc.).

(4) S is the abbreviation of Interruptible Sleep, that is, interruptible sleep, indicating that the process is suspended by the system because it is waiting for an event. When the event that the process is waiting for occurs, it is awakened and enters the R state.

(5) I is the abbreviation for Idle, or idle state, and is used on kernel threads that do not interrupt sleep. As mentioned earlier, non-interruptible processes caused by hardware interactions are represented by D, but for some kernel threads, they may not actually have any load, and Idle is used to distinguish this situation. Note that processes in D state lead to an increase in average load, while processes in I state do not.

There are two unusual states of a process:

(1) T or t, which stands for Stopped or Traced, indicates that the process is in a paused or tracked state.

Send a SIGSTOP signal to a process, it will become Stopped in response to this signal; then send a SIGCONT signal to it, and the process will resume running (if the process is started directly in the terminal, you need to use the fg command to resume running in the foreground).

When you debug a process with a debugger (such as gdb), after interrupting the process with a breakpoint, the process becomes tracked, which is actually a special paused state, but you can use the debugger to track and control the running of the process as needed.

(2) X, which stands for Dead, indicates that the process is dead, so you won't see it in the top or ps commands.

Note that ⚠️: Ss+:S indicates interruptible sleep, s indicates that the process is the leader of a session, and + indicates the foreground process group.

Process groups and sessions:

They manage a set of interrelated processes, meaning as follows:

(1) A process group represents a group of interrelated processes, for example, each child process is a member of the group of the parent process.

(2) session refers to one or more process groups that share the same control terminal.

For example, if we log in to the server through SSH, we will open a control terminal (TTY) that corresponds to a session. The commands we run in the terminal and their child processes constitute process groups one by one, in which the commands running in the background constitute the background process group, and the commands running in the foreground constitute the foreground process group.

Install dstat

(1) CentOS:yum install-y dstat

(2) Ubuntu:apt install dstat

Here dstat is a new performance tool, which absorbs the advantages of several tools such as vmstat, iostat, ifstat, etc., and can observe the system CPU, disk Imax O, network and memory usage at the same time.

Scenario 8: what if there are a large number of uninterruptible and zombie processes in the system (part two)

Tool usage:

(1) dstat 1 10 (output 10 sets of data at intervals of 1 second)

Linux performance optimization reading notes 01

(2) pidstat-d-p 4344 1 3 (- d displays the statistical data of Icano,-p specifies the process number, and outputs 3 sets of data at intervals of 1 second)

(3) strace-p 6082 (- p specifies the process number)

Strace is the most commonly used tool for tracking process system calls.

(4) perf record-g (the terminal runs for about 15 minutes, then ctrl+c)

(5) perf report

Handling of zombie processes

To solve the zombie process, we need to find its root, that is, to find the parent process, and then solve it in the parent process.

(1) pstree-aps 3084 (- a represents the output command line option, p represents the PID,s represents the parent process of the specified process)

Linux performance optimization reading notes 01

After running, you will find that the parent process of process 3084 is 4009, that is, the app application.

(2) then look at the code of the app application to see if the end of the child process is handled correctly, such as calling wait () or waitpid (), or registering the handler for the SIGCHLD signal.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.