Shulou(Shulou.com)06/01 Report--

View performance order:

[cpu] mpstat-P ALL 1 100 (sar-uther sar-p)

[network] sar-n DEV

[disk] sar-bMagazine sar-d

[mem] sar-WBJ sar-RJE sar-B

Top

Historical data View:

Ubuntu

Sar / var/log/sysstat/sa

Centos

Sar / var/log/sa/sa12 [1-31]

Optimization process:

Cpu

Root@python-develpment:~# cat / proc/cpuinfo | less

Processor: 0

Vendor_id: GenuineIntel

Cpu family: 6

Model: 42

Model name: Intel (R) Core (TM) i3-2130 CPU @ 3.40GHz

Stepping: 7

Microcode: 0x26

Cpu MHz: 3399.155

Cache size: 3072 KB

If there is a big difference in the red part marked above, you should turn off the cpu power management (adjust it in bios)

2. Mem

Root@python-develpment:~# cat / proc/meminfo

MemTotal: 3763212 kB

MemFree: 2071820 kB

MemAvailable: 2753332 kB

It's best to turn off numa

Modify the boot kernel load file to add numa=off

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a / boot partition. This means that

# all kernel and initrd paths are relative to / boot/, eg.

# root (hd0,0)

# kernel / vmlinuz-version ro root=/dev/sda2

# initrd / initrd- [generic-] version.img

# boot=/dev/sda

Default=0

Timeout=5

Splashp_w_picpath= (hd0,0) / grub/splash.xpm.gz

Hiddenmenu

Title CentOS (2.6.32-220.el6.x86_64)

Root (hd0,0)

Kernel / vmlinuz-2.6.32-220.el6.x86_64 ro root=UUID=b9e7e53c-e172-479c-aa4c-28c72492a5b6 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=

Latarcyrheb-sun16 rhgb crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM numa=off

Initrd / initramfs-2.6.32-220.el6.x86_64.img

~

Or:

Start the database with numa parameters (to prevent memory leaks)

[root@mysql-master] # numactl-- interleave=nodes mysqld

Close swapiness (database sets it to 0, memory database closes it, swapoff)

[root@mysql-master ~] # sysctl-a | grep swap

Vm.swappiness = 60

Vi / etc/sysctl.conf

Add: vm.swappiness = 0

Execute: sysctl-p to make it effective

The bigger the better.

3. Io scheduling

{centos}

[root@mysql-master ~] # cat / sys/block/sdb/queue/scheduler

Noop anticipatory deadline [cfq]

[root@mysql-master ~] #

{ubuntu}

Root@python-develpment:~# cat / sys/block/sdb/queue/scheduler

Noop [deadline] cfq

Summary

CPU: turn off power protection mode

Memory: (MySQL database) vm.swappiness = 0, only memory database (redis,memcached) closes swap:swapoff, closes numa

File system: the system is mounted with noatime,nobarrier, and the IO scheduling policy is modified to deadline.

-mpstat--

Detailed explanation of Linux CPU Real-time Monitoring mpstat Command

Brief introduction

Mpstat, the abbreviation of Multiprocessor Statistics, is a real-time system monitoring tool. It reports along with some statistics for CPU, which are stored in the / proc/stat file. In a multi-CPUs system, it can view not only the average status information of all CPU, but also the information of a specific CPU. The most important feature of mpstat is that it can view the statistics of each computing core in the multi-core cpu, while the similar tool vmstat can only view the overall cpu of the system.

Syntax mpstat [- P {| ALL}] [internal [count]] parameter explanation-P {| ALL} indicates which CPU to monitor. Cpu values the interval between two samples adjacent to internal and the number of count samples in [0 CPU count-1]. Count can only be used with delay. When there are no parameters, mpstat displays the average of all information after the system starts. With interval, the average information in the first line since the system was started. Starting from the second line, the output is the average information of the previous interval period. Example

View the current health information of the multicore CPU core and update it every 2 seconds

Mpstat 219nice% sys% iowait% irq% soft% steal% guest% idle19:45:14 all 0.04 0.00 0.00 0.00 99.9619 all 0.00 0.00 0.03 0.00 0.00 0.00 99.97 19V 45 all 0.00 0.07 0.00 0.00 0.00 99.87

If you want to see detailed current health information for each cpu core, the output is as follows:

Mpstat-P ALL 219 CPU% nice% sys% iowait% soft% steal% guest% idle19:43:59 all 0.00 0.00 0.04 0.00 0.00 0.00 99.9619 00 0.00 100.0019:43:59 1 0.00 0.00 0.00 100.00.19:43:59 13 0.99 0.00 0.00 0.00 99.0119:43:59 14 0.00 0.00 0. 00 0.00 0.00 0.00 100.0019:43:59 15 0.00 0.00 0.00 100.00

The field has the following meaning

% user in the internal period, the CPU time in user mode (%), excluding the negative process (usr/total) * 100%nice in the internal period, the CPU time (%) (nice/total) * 100%sys in the internal period, kernel time (%) (system/total) * 100%iowait in the internal period Hard disk IO wait time (%) (iowait/total) * 100%irq in internal period, hard interrupt time (%) (irq/total) * 100%soft in internal period, soft interrupt time (%) (softirq/total) * 100%idle in internal period, CPU idle time for any reason except waiting for disk IO operation (%) (idle/total) * 100

The calculation formula is as follows

Total_cur=user+system+nice+idle+iowait+irq+softirqtotal_pre=pre_user+ pre_system+ pre_nice+ pre_idle+ pre_iowait+ pre_irq+ pre_softirquser=user_cur-user_pretotal=total_cur-total_pre where _ cur represents the current value and _ pre represents the value before the interval time. All values in the above table can be taken to two decimal places.

-- sar--

Sar (System Activity Reporter system activity report) is one of the most comprehensive system performance analysis tools on Linux, which can report system activities from many aspects, including: reading and writing of files, usage of system calls, disk ID O, CPU efficiency, memory usage, process activities and IPC-related activities. This paper mainly introduces the sar command by taking CentOS 6.3x64 system as an example.

Common format of sar command

Sar [options] [- A] [- o file] t [n]

Where:

For example: sar-d 3 10

T (3) is the sampling interval, n (10) is the sampling times, and the default value is 1.

-o file means to store the result of the command in a file in binary format, where file is the file name.

Options is a command line option, and the common options for sar commands are as follows:

-A: the sum of all reports

-u: output statistics on CPU usage

-v: output statistics for inode, files, and other kernel tables

-d: output activity information for each block device

-r: output memory and swap space statistics

-b: displays statistics for Ihammer O and transfer rate

-a: reading and writing of documents

-c: output process statistics, number of processes created per second

-R: output statistics for memory pages

-y: terminal device activity

-w: output system exchange activity information

1. CPU resource monitoring

For example, to sample every 10 seconds, for 3 consecutive times, to observe the usage of CPU, and to store the sampling results in the file test in the current directory in binary form, type the following command:

Sar-u-o test 10 3

The screen displays as follows:

17:06:16 CPU user nice system iowait steal idle

17:06:26 all 0.00 0.00 0.20 0.00 0.00 99.80

17:06:36 all 0.00 0.00 0.20 0.00 0.00 99.80

17:06:46 all 0.00 0.00 0.10 0.00 0.00 99.90

Average: all 0.00 0.00 0.17 0.00 0.00 99.83

Output item description:

CPU:all indicates that the statistics are the average of all CPU.

% user: displays the percentage of total time spent using CPU running at the user level (application).

% nice: the percentage of total CPU time spent on nice operations displayed at the user level.

% system: the percentage of total time spent running CPU at the core level (kernel).

% iowait: displays the percentage of total CPU time spent waiting for the IUnip O operation.

% steal: the percentage that the hypervisor (hypervisor) waits for the virtual CPU while serving another virtual process.

% idle: displays the percentage of total CPU time occupied by CPU idle time.

1. If the value of% iowait is too high, it indicates that there is a bottleneck in the hard disk.

two。 If the value of% idle is high but the system response is slow, it is possible that CPU is waiting to allocate memory. At this time, the memory capacity should be increased.

3. If the value of% idle is consistently lower than 1, the CPU processing capacity of the system is relatively low, indicating that the resource that needs to be solved most in the system is CPU.

To view the contents of the binary file test, type the following sar command:

Sar-u-f test

2. Inode, file and other kernel table monitoring

For example, to observe the status of the core table by sampling every 10 seconds for 3 consecutive times, type the following command:

Sar-v 10 3

The screen displays as follows:

17:10:49 dentunusd file-nr inode-nr pty-nr

17:10:59 6301 5664 12037 4

17:11:09 6301 5664 12037 4

17:11:19 6301 5664 12037 4

Average: 6301 5664 12037 4

Output item description:

Dentunusd: the number of unused entries in the directory cache

File-nr: number of file handles (file handle) used

Inode-nr: number of index node handles (inode handle) used

Pty-nr: number of pty used

3. Memory and swap space monitoring

For example, sample every 10 seconds, sample 3 times in a row, and monitor memory paging:

Sar-r 10 3

The screen displays as follows:

Output item description:

Kbmemfree: this value is basically the same as the free value in the free command, so it does not include buffer and cache space.

Kbmemused: this value is basically the same as the used value in the free command, so it includes buffer and cache space.

% memused: this value is a percentage of kbmemused and total memory (excluding swap).

Kbbuffers and kbcached: these two values are buffer and cache in the free command.

Kbcommit: guarantee the memory needed by the current system, that is, the memory needed to ensure that it does not overflow (RAM+swap).

% commit: this value is a percentage of kbcommit to total memory (including swap).

4. Memory paging monitoring

For example, sample every 10 seconds, sample 3 times in a row, and monitor memory paging:

Sar-B 10 3

The screen displays as follows:

Output item description:

Pgpgin/s: indicates the number of bytes per second from disk or SWAP to memory (KB)

Pgpgout/s: indicates the number of bytes per second from memory to disk or SWAP (KB)

Fault/s: the number of missing pages generated by the system per second, that is, the sum of primary and secondary missing pages (major + minor)

Majflt/s: the number of main missing pages generated per second.

Pgfree/s: the number of pages placed in the idle queue per second

Pgscank/s: number of pages scanned by kswapd per second

Pgscand/s: the number of pages directly scanned per second

Pgsteal/s: the number of pages cleared from cache per second to meet memory requirements

% vmeff: percentage of pages cleared per second (pgsteal) to total scanned pages (pgscank+pgscand)

5. Icano and transfer rate monitoring

For example, to report buffer usage by sampling every 10 seconds for 3 consecutive times, type the following command:

Sar-b 10 3

The screen displays as follows:

18:51:05 tps rtps wtps bread/s bwrtn/s

18:51:15 0.00 0.00 0.00

18:51:25 1.92 0.00 1.92 0.00 22.65

18:51:35 0.00 0.00 0.00

Average: 0.64 0.00 0.64 0.00 7.59

Output item description:

Tps: the total amount of Istroke O transmissions per second of physical devices

Rtps: the total amount of data read from the physical device per second

Wtps: the total amount of data written to the physical device per second

Bread/s: the amount of data read from a physical device per second, in blocks per second

Bwrtn/s: the amount of data written to the physical device per second, in blocks per second

6. Monitoring of process queue length and average load status

For example, sample every 10 seconds, sample 3 times in succession, and monitor the queue length and average load status of the process:

Sar-Q 10 3

The screen displays as follows:

19:25:50 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

19:26:00 0 259 0.00 0.00 0.00

19:26:10 0 259 0.00 0.00 0.00

19:26:20 0 259 0.00 0.00 0.00

Average: 0 259 0.00 0.00 0.00

Output item description:

Runq-sz: length of the run queue (number of processes waiting to run)

Plist-sz: number of processes (processes) and threads (threads) in the process list

Ldavg-1: average system load in the last minute (System load average)

Ldavg-5: average system load over the past 5 minutes

Ldavg-15: average system load over the past 15 minutes

7. System exchange activity information monitoring

For example, sampling every 10 seconds, sampling 3 times in a row, and the monitoring system exchanges activity information:

Sar-W 10 3

The screen displays as follows:

19:39:50 pswpin/s pswpout/s

19:40:00 0.00 0.00

19:40:10 0.00 0.00

19:40:20 0.00 0.00

Average: 0.00 0.00

Output item description:

Pswpin/s: the number of swap pages (swap page) swapped in by the system per second

Pswpout/s: the number of swap pages (swap page) swapped out by the system per second

8. Equipment usage monitoring

For example, to report device usage by sampling every 10 seconds for 3 consecutive times, type the following command:

# sar-d 10 3-p

The screen displays as follows:

17:45:54 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm util

17:46:04 scd0 0.00 0.00 0.00

17:46:04 sda 0.00 0.00 0.00

17:46:04 vg_livedvd-lv_root 0.00 0.00 0.00

17:46:04 vg_livedvd-lv_swap 0.00 0.00 0.00

Where:

The parameter-p can print out the disk device names such as sda,hdc. If the parameter-p is not used, the device node may be dev8-0dev22-0.

Tps: the number of times per second from the physical disk. Multiple logical requests will be merged into a single disk request, and the size of a transfer is uncertain.

Rd_sec/s: the number of times sectors are read per second.

Wr_sec/s: the number of times sectors are written per second.

Avgrq-sz: the average data size (sector) of each device Istroke O operation.

Avgqu-sz: the average length of the disk request queue.

Await: the average elapsed time of each request, including the waiting time of the request queue, from the request disk operation to the completion of the system processing, in milliseconds (1 second = 1000 milliseconds).

Svctm: the average time the system processes each request, excluding the time spent in the request queue.

% util:I/O requests as a percentage of CPU, the higher the ratio, the more saturated it is.

1. When the value of avgqu-sz is low, the utilization of the device is higher.

two。 When the value of% util is close to 1%, the device bandwidth is full.

To determine the bottleneck of the system, it sometimes requires a combination of several sar command options

It is suspected that there is a bottleneck in CPU. You can check it with sar-u and sar-Q.

Memory bottleneck is suspected. Check it with sar-B, sar-r, sar-W, etc.

It is suspected that there is a bottleneck in IBG O, which can be checked by sar-b, sar-u and sar-d.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.