Shulou(Shulou.com)06/01 Report--

Editor to share with you how to achieve Linux performance monitoring, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

The TOP command shows that some Oracle session take up too much CPU resources. The sad thing is that you can't even get into sqlplus on the server, and there's no response to the command. I had to restart the server. After rebooting, I looked at it again and found that there was a problem with the SQL tested by a colleague. A SQL takes up 30% of the CPU.

In the study of this issue, google also came to some monitoring items under Linux, sorted out as follows.

one。 Overview of Linux performance monitorin

The system consists of several subsystems. Usually, modifying one subsystem may affect another subsystem, and even lead to the instability and collapse of the whole system. So optimization, monitoring, and testing are usually linked together, and it is a cyclic and long-term process, and the subsystems that are usually monitored are as follows:

(1)。 CPU

(2)。 Memory

(3)。 IO

(4)。 Network

These subsystems depend on each other, so it is very helpful to understand the characteristics of these subsystems, monitor the performance parameters of these subsystems and find possible bottlenecks in time.

1.1 Application Typ

Different systems are used for different purposes. To find the performance bottleneck, you need to know what applications the system is running and what characteristics the system is running. For example, the requirements of web server for the system are definitely different from those of file server, so it is important to distinguish the application types of different systems. Generally, applications can be divided into two types:

(1) IO-related, IO-related applications are usually used to deal with a large amount of data, requiring a lot of memory and storage, frequent IO operations to read and write data, but less requirements for CPU, most of the time CPU is waiting for the hard disk, such as database server, file server and so on.

(2) CPU-related, CPU-related applications need to use a lot of CPU, such as highly concurrent web/mail server, image / video processing, scientific computing and so on can be regarded as CPU-related applications.

Looking at the actual example, the first is the characteristics shown by the file server when copying a large file:

$vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 4140 335516 4852308 4852308 65024 1442 563 02 47 52 004 140 1961816 335516 4853868 0768 65536 1434 522 0 150 48 00 4 140 1960788 335516 4855300 00 768 48640 1412 573 0 150 49 00 4 140 1958528 335516 4857280 00 1024 65536 1415 521 0 141 57 00 51 40 1957488 335516 4858884 00 768 81412 1504 609 02 50 49 0

The second is the characteristic that CPU shows when doing a lot of calculations:

$vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 40 3625096 334256 3266584 00 16 1054 470 00 00 40 3625220 334264 3266576 00 0 12 1037 448 100 00 00 40 140 3624468 334264 3266580 00 0 148 1160 632 100 00 00 40 140 3624468 334264 3266580 00 00 1078 527 100 00 00 40 140 3624712 334264 3266580 00 0 80 1053 501 100 00 00

The most obvious difference between the above two examples is the id column, which represents the free rate of CPU. The id is maintained at about 50% when copying files, and the id is basically 0 when CPU is calculated in large quantities.

1.2 bottom line

Establish a bottom line in advance, if the statistics from performance monitoring cross this line, we can say that the performance of the system is poor, and if the data can be kept online, we can say that the performance is good. Establishing such a bottom line requires some theory, additional load testing, and years of experience as a system administrator. If you don't have many years of experience, a simple way to draw a bottom line is to build this bottom line on your expectations of the system. What kind of performance do you expect this system to have? this is the bottom line. If you do not meet this requirement, the performance is poor.

1.3 Monitoring tools

Tools

A brief introduction

Top

Check the process activity status and some system status

Vmstat

View system status, hardware, system information, etc.

Iostat

Check CPU load, hard drive status

Sar

Integrated tools to view system status

Mpstat

View multiprocessor status

Netstat

View the state of the network

Iptraf

Real-time network condition monitoring

Tcpdump

Grab network packets and analyze them in detail

Mpstat

View multiprocessor status

Tcptrace

Packet analysis tool

Netperf

Network bandwidth tool

Dstat

Integrated tool, integrated vmstat, iostat, ifstat, netstat and other information

two。 CPU

The occupation of CPU mainly depends on what kind of resources are running on CPU. For example, copying a file usually takes less CPU, because most of the work is done by DMA (Direct Memory Access), but after completing the copy, give an interrupt to let CPU know that the copy has been completed; scientific computing usually takes up more CPU, most of the computing work needs to be completed on CPU, memory, hard disk and other subsystems only do temporary data storage work. To monitor and understand the performance of CPU, you need to know some basic knowledge of the operating system, such as interrupts, process scheduling, process context switching, runnable queues and so on. Here is an example to briefly introduce these concepts and their relationship. CPU has work to do all the time (processes, threads) and has its own to-do list (runnable queue). It is up to the boss (process scheduling) to decide what he should do. He needs to communicate with the boss in order to get his boss's ideas and adjust his work in time (context switch). After some of the work is done, you still need to report to the boss in time, so in addition to doing what you are supposed to do, wage earners (CPU) also spend a lot of time and energy on communication and reporting.

CPU is also a kind of hardware resource, which needs driver and manager just like any other hardware device. We can regard the process scheduling of kernel as the management program of CPU, which is used to manage and allocate CPU resources, reasonably arrange processes to preempt CPU, and decide which process should use CPU and which process should wait. Process scheduling in the operating system kernel is mainly used to schedule two types of resources: processes (or threads) and interrupts. Process scheduling assigns different priorities to different resources, the highest priority is hardware interrupts, followed by kernel (system) processes, and finally user processes. Each CPU maintains a runnable queue for runnable threads. The thread is either asleep (blocked is waiting for IO) or runnable. If the current load of CPU is too high and new requests continue, the process scheduling will be unable to cope temporarily. At this time, the thread will have to be temporarily placed in the runnable queue.

You can monitor CPU information from the following aspects:

(1) interrupt

(2) context switching

(3) runnable queue

(4) CPU utilization.

2.1 bottom line

Usually we expect our system to achieve the following goals:

(1) CPU utilization, if CPU has 100% utilization, then a balance should be reached: 65% User Time,30%-35% System Time,0%-5% Idle Time 70%

(2) context switching. Context switching should be related to CPU utilization. If the above CPU utilization balance can be maintained, a large number of context switching is acceptable.

(3) runnable queues, each runnable queue should not have more than 3 threads (per processor). For example, there should be no more than 6 threads in the runnable queue of a dual-processor system.

2.2 vmstat

Vmstat is a gadget to view the overall performance of the system. It is small, works well even in very heavy situations, and can collect continuous performance data at time intervals.

$vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 1140 336304 3531996 3531996 1166 1166 5033 33 70 25 001 2788296 336304 3531996 0 0 0 0 1194 5605 33 69 25 0 0 1 140 2788436 336304 3531996 0 0 0 1249 8036 5 4 67 25 0 0 1 140 2782688 336304 3531996 0 0 0 1333 7792 6 6 64 25 03 1 140 2779292 336304 3531992 0 0 0 28 1323 7087 4 5 67 25 0

Parameter description:

(1)。 R, the number of threads that can run the queue, all of which are runnable, but CPU is temporarily unavailable

(2)。 B, the number of processes being blocked, waiting for IO request

(3)。 In, number of interrupts handled

(4)。 Cs, the number of context switches being done on the system

(5)。 Us, percentage of CPU occupied by users

(6)。 Percentage of CPU occupied by sys, kernel and interrupts

(7)。 Wa, the percentage of idle CPU after all runnable threads are waiting for IO after being blocked

(8)。 Percentage of id,CPU completely idle

Let's take two real-life examples to actually analyze:

$vmstat 1procs-memory--swap---io-----system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 40 2915476 341288 3951700 00 1057 523 19 81 00 40 2915724 341296 3951700 00 0 1048 546 19 81 00 04 0 140 2915848 341296 3951700 00 00 1044 514 18 82 00 04 0 140 2915848 341296 3951700 00 0 24 1044 564 20 80 00 04 0 140 2915848 341296 3951700 00 00 1060 546 18 82 00 0

From the above data, we can see a few points:

(1)。 The interrupts (in) is very high, while the context switch (cs) is relatively low, indicating that this CPU is constantly requesting resources.

(2)。 User time (us) has been kept above 80%, and context switching is low (cs), indicating that a process may have been occupying CPU.

(3)。 Run queue (r) happens to be four.

$vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st14 0140 2904316 341912 3952308 000460 1106 9593 36 64 100 017 0140 2903492 341912 39517800 0 00 1037 9614 35 65 10 020 0 140 2902016 341912 3952000 000 0 1046 9739 35 64 10 017 0 140 2903904 341912 3951888 000 76 1044 9879 37 63 00 016 0 140 2904580 341912 3952108 000 0 1055 9808 34 65 10 0

From the above data, we can see a few points:

(1)。 Context switch (cs) is much higher than interrupts (in), indicating that the kernel has to switch processes back and forth.

(2)。 Further observation shows that system time (sy) is very high and user time (us) is very low, coupled with high frequency context switching (cs), indicating that the running application calls a large number of system calls (system call).

(3)。 Run queue (r) is more than 14 threads, according to the hardware configuration of this test machine (quad core), it should be kept within 12.

My message at 100% CPU in the morning:

Top-11:49:08 up 50 days, 22:25, 6 users, load average: 59.79,59.98,60.50

Tasks: 200 total, 61 running, 139 sleeping, 0 stopped, 0 zombie

Cpu0: 26.5%us, 73.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu1: 25.0%us, 75.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 1939780k total, 1744412k used, 195368k free, 95704k buffers

Swap: 4401800k total, 662836k used, 3738964k free, 811124k cached

[root@localhost ~] # vmstat 2 10

Procs-memory--swap---io-----system---cpu-

R b swpd free buff cache si so bi bo in cs us sy id wa st

58 1 662836 195988 95428 810740 0 0 4 106 4 1 23 4 72 10

59 1 662836 195988 95448 810732 0 0 0 128 235 221 28 72 0 0 0

59 1 662836 195988 95448 810768 0 0 0 216 209 28 72 0 0 0

2.3 mpstat

Mpstat is similar to vmstat, except that mpstat can output data from multiple processors.

Note: you need to install the sysstat package before you have this command. You can install it using yum:

# yum install sysstat

Sysstat contains iostat, mpstat, sar, commands.

[root@localhost gmail] # export LANG=en_ us [root @ localhost gmail] # mpstat-P ALLLinux 2.6.18-8.el5xen (localhost.localdomain) 02Greater 2011

10:20:16 PM CPU% user% nice% sys% iowait% irq% soft% steal% idle intr/s10:20:16 PM all 11.49 0.00 2.58 1.04 0.01 0.13 0.01 84.74 314.6110 V 16 PM 01 5.73 0.00 2.56 0.55 0.02 0.23 0.01 80.89 241. 0910 PM 20 PM 1 7.25 0.00 2.60 1.53 0.00 0.02 0.01 88.59 73.52 [root@localhost gmail] # mpstat-P ALL 1Linux 2.6.18-8.el5xen (localhost.localdomain) 02 Unix 2011

10:20:18 PM CPU% user% nice% sys% iowait% irq% soft% steal% idle intr/s10:20:19 PM all 0.00 0.00 0.00 100.00 136.6310 nice 20 19 PM 00.00 0.00 0.00 0.00 100.00 86.1410 : 20:19 PM 1 0.00 0.00 0.00 100.00 50.50

10:20:19 PM CPU% user% nice% sys% iowait% irq% soft% steal% idle intr/s10:20:20 PM all 0.00 0.00 0.00 0.47 0.00 0.00 0.00 99.53 105.0010 user 20 PM 00.00 0.00 0.00 0.00 100.00 79. 0010:20:20 PM 1 0.00 0.00 0.00 0.90 0.00 0.00 0.00 99.10 26.00

2.4 ps

Check how much CPU resources are consumed by a program and process:

[root@localhost gmail] # while:; do ps-eo pid,ni,pri,pcpu,psr,comm | grep 'oracle'; sleep 1 Done PID NI PRI% CPU PSR COMMAND 3668 0 24 00 0 oracle 3670 0 21 00 0 oracle 3672 0 24 0 0 0 oracle 3674 0 23 00 0 oracle 3676 0 24 0.0 1 oracle 3678 0 23 0 0 0 oracle 3680 0 21 0.0 1 oracle 3682 0 24 0.0 1 oracle 3684 0 24 0 0 0 oracle 3686 0 21 00 oracle

three。 Memory

The "memory" mentioned here includes physical memory and virtual memory. Virtual memory (Virtual Memory) extends the computer's memory space to the hard disk. Physical memory (RAM) and part of the hard disk space (SWAP) are combined as virtual memory to provide a coherent virtual memory space for the computer. The advantage is that we have "more memory" and can run more and larger programs. The disadvantage is that the overall performance of using part of the hard disk as memory is affected, the read and write speed of the hard disk is several orders of magnitude slower than the memory, and the exchange between RAM and SWAP increases the burden of the system.

In the operating system, virtual memory is divided into pages, and on x86 systems, each page size is 4KB. The Linux kernel reads and writes virtual memory in "pages". It is page-by-page when memory is transferred to the hard disk swap space (SWAP) and read from swap space to memory. This exchange process of memory and SWAP is called page exchange (Paging), it is worth noting that paging and swapping are two completely different concepts, many domestic reference books confuse these two concepts, swapping is also translated into exchange, in the operating system refers to a program completely swapped to the hard disk to free memory for the new program to use, and paging only exchanges the part of the program (page) is two different concepts. Pure swapping has been difficult to see in modern operating systems, because the method of swapping the whole program to the hard disk is time-consuming, laborious and unnecessary. Modern operating systems are basically a mixture of paging or paging/swapping, and swapping was originally implemented on Unix system V.

Only two kernel processes related to performance monitoring, kswapd and pdflush, are introduced here.

(1) kswapd daemon is used to check pages_high and pages_low, start scanning and try to free 32 pages if the available memory is less than pages_low,kswapd, and repeat the process of scanning and releasing until the available memory is greater than pages_high.

Check 3 things when scanning:

1) if the page has not been modified, put the page in the list of available memory

2) if the page is modified by the file system, write the content of the page to disk

3) if the page is modified, but not by the file system, write the page to swap space.

(2) pdflush daemon is used to synchronize the memory pages related to files and synchronize the memory pages to the hard disk in time. For example, after opening a file, the file is imported into memory, and after the file is modified and saved, the kernel does not immediately save the file to the hard disk. It is up to pdflush to decide when to write the corresponding page to the hard disk, which is controlled by a kernel parameter vm.dirty_background_ratio. For example, the following parameter shows that the dirty page (dirty pages) starts to be written to the hard disk when the dirty page reaches 10% of all memory pages.

# / sbin/sysctl-n vm.dirty_background_ratio

ten

3.1 vmstat

# vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 3 252696 2432 268 7148 3604 2368 3608 2372 288 288 0 21 78 10 2 253484 2216 228 7104 5368 2976 5372 3036 930 519 000 100 00 1 259252 2616 128 6148 19784 18712 19784 18712 3821 1853 0 13 95 11 2 260008 2188 144 6824 11824 2584 12664 2584 1347 1174 140 0 86 0 21 262140 2964 128 5852 24912 17304 24952 17304 4737 2341 86 100 04

Part of the parameter description:

(1)。 Swpd, the size of the SWAP space used, in KB

(2)。 Free, the amount of physical memory available, in KB

(3)。 Buff, the buffer size of physical memory used to cache read and write operations, in KB units

(4)。 Cache, physical memory is used to cache the cache size of the process address space, in KB units

(5)。 Si, the size of the data read from SWAP to RAM (swap in), in KB

(6)。 So, the size of data written from RAM to SWAP (swap out), in KB

(7)。 Bi, the size of the disk block read from the file system or SWAP to RAM (blocks in), in block units

(8)。 Bo, the size of a block written from RAM to a file system or SWAP (blocks out), in block units

The above is an example of a frequent read-write swap area, and the following points can be observed:

(1)。 There is basically no significant change in physical available memory free, but swapd increases gradually, indicating that the minimum available memory is always maintained at 256MB (physical memory size) * 10% = 2.56MB. When dirty pages reach 10% (vm.dirty_background_ratio = 10), swap is widely used.

(2)。 The steady decrease in buff indicates that the system knows that there is not enough memory, and kwapd is borrowing some memory from buff.

(3)。 Kswapd continues to write dirty pages to the swap exchange (so), and it can be seen from the gradual increase in swapd. According to the three things checked during the kswapd scan mentioned above, if the page is modified, but not by the file system, write the page to swap, so here swapd continues to increase.

four。 IO

The disk is usually the slowest subsystem of the computer, and it is also the place where performance bottlenecks are most likely to occur, because the disk is farthest away from the CPU and CPU access to the disk involves mechanical operations, such as axis rotation, orbit finding, and so on. The speed difference between accessing the hard disk and accessing memory is calculated by an order of magnitude, just like the difference between a day and a minute. To monitor IO performance, it is necessary to understand the fundamentals and how Linux handles IO between hard disk and memory.

4.1 memory page

In the third section of Memory, it is mentioned that the IO between the memory and the hard disk is carried out in pages, and the size of 1 page on the Linux system is 4K. You can view the default page size of the system with the following command:

$/ usr/bin/time-v date

...

Page size (bytes): 4096

...

4.2 missing page interrupt

Linux uses virtual memory to greatly expand the program address space, so that programs that cannot accommodate the original physical memory can also gain more memory by constantly swapping the temporarily unused memory pages to the hard disk and reading the needed memory pages from the hard disk to the hard disk, as if the physical memory has been expanded. In fact, this process is completely transparent to the program, regardless of which part of itself and when it is swapped into memory, everything is accomplished by the virtual memory management of the kernel. When the program starts, the Linux kernel first checks the CPU cache and physical memory, ignores it if the data is already in memory, causes a page fault (Page Fault) if the data is not in memory, then reads the page fault from the hard disk and caches the page fault into physical memory. The page fault interrupt can be divided into primary page fault interrupt (Major Page Fault) and secondary page fault interrupt (Minor Page Fault). The interrupt caused by reading data from disk is the main page fault interrupt; the data has been read into memory and cached, and the interrupt caused by reading data from the memory cache rather than directly from the hard disk is the secondary page fault interrupt.

The above memory cache plays the role of pre-reading the hard disk. The kernel first looks for missing pages in physical memory, then generates a secondary page fault interrupt from the memory cache, and reads it from the hard disk if it has not been found. Obviously, taking the extra memory out to make a memory cache improves access speed, and there is also a hit rate problem, which, with luck, will greatly improve performance if you can read from the memory cache every time a page is missing. A simple way to improve the hit ratio is to increase the memory cache area. The larger the cache area, the more pages are stored, and the higher the hit ratio will be.

The following time command can be used to see how many primary and secondary page faults occur when a program starts for the first time:

$/ usr/bin/time-v date... Major (requiring I reclaiming a frame O) page faults: 1 Minor (reclaiming a frame) page faults: 260.

4.3 File Buffer Cache

Reading pages from the above memory cache (also known as file cache File Buffer Cache) is much faster than reading pages from the hard disk, so the Linux kernel wants to generate secondary page fault interruptions (read from the file cache) as much as possible, and avoid primary page interruptions (read from the hard disk) as much as possible, so that with the increase of secondary page fault interruptions, the file cache area will gradually increase. It wasn't until the system had only a small amount of physical memory available that Linux began to release some unused pages. After running Linux for a period of time, we will find that although there are not many programs running on the system, there is always little available memory, which gives us the illusion that Linux is very inefficient in memory management. in fact, Linux uses those temporarily unused physical memory efficiently for pre-storage (memory cache). Here is the physical memory and file cache on a Sun server:

$cat / proc/meminfoMemTotal: 8182776 kBMemFree: 3053808 kBBuffers: 342704 kBCached: 3972748 kB

This server has a total of 8GB physical memory (MemTotal), 3GB around available memory (MemFree), 343MB around for disk cache (Buffers), and 4GB around for file cache (Cached). It can be seen that Linux really uses a lot of physical memory for Cache, and this cache area can continue to grow.

4.4 Page Type

There are three types of memory pages in Linux:

(1)。 Read pages, read-only pages (or code pages), those pages that are read from the hard disk through the main missing page interrupt, including static files, executable files, library files, etc., that cannot be modified. When the kernel needs them, it reads them into memory, when it runs out of memory, the kernel releases them to the free list, and when the program needs them again, it needs to read them again through a page fault interrupt.

(2)。 Dirty pages, dirty pages, refer to those data pages that have been modified in memory, such as text files, etc. Pdflush is responsible for synchronizing these files to the hard disk, and when there is insufficient memory, kswapd and pdflush write the data back to the hard disk and release memory.

(3)。 Anonymous pages, anonymous pages, those that belong to a process but are not associated with any files, cannot be synchronized to the hard disk, and kswapd is responsible for writing them to the swap partition and freeing memory when there is insufficient memory.

4.5 IO's Per Second (IOPS)

Each disk IO request takes a certain amount of time, and the wait time is simply unbearable compared to accessing memory. On a typical 1GHz PC in 2001, it takes 8000000 nanosec = 8 millisec for random disk access to a word, 200 nanosec; for sequential access to a word, and only 10 nanosec to access a word from memory. (data from: Teach Yourself Programming in Ten Years) this hard disk can provide 125x IOPS (1000 ms / 8 ms).

4.6 Sequential IO and Random IO

IO can be divided into sequential IO and random IO. Before performance monitoring, it is necessary to find out whether the system is biased towards sequential IO or random IO.

(1) Sequential IO means that a large amount of data is requested sequentially at the same time, such as database performing a large number of queries, streaming media services, etc. Sequential IO can move a large amount of data quickly at the same time. The performance of IOPS can be evaluated by dividing the number of read and write IO bytes per second by the number of read and write IOPS per second, and rkB/s divided by r _ peg _ r _ r _ B _ b _ b _ s divided by w _ bank s. The following shows the IO for 2 consecutive seconds, which shows that the data written by IO increases each time (45060.00 / 99.00 = 455.15 KB per IO,54272.00 / 112.00 = 484.57 KB per IO).

Compared with random IO, sequential IO should pay more attention to the throughput capacity (KB per IO) of each IO:

Iostat-kx 1avg-cpu:% user% nice% system% iowait% steal% idle 0.00 0.00 2.50 25.0.00 72.25

Device: rrqm/s wrqm/s rUnip s wbank s rkB/s wkB/s avgrq-sz avgqu-sz await svctm% utilsdb 24.00 19995.00 29.00 99.00 4228.00 45060.00 770.12 45.01 539.65 7.80 99.80

Avg-cpu:% user% nice% system% iowait% steal% idle 0.00 0.00 1.00 30.67 0.00 68.33

Device: rrqm/s wrqm/s ramp s wbank s rkB/s wkB/s avgrq-sz avgqu-sz await svctm% utilsdb 3.00 12235.00 3.00 112.00 768.00 54272.00 957.22 144.85 576.44 8.70 100.10

(2) Random IO refers to randomly requested data. Its IO speed does not depend on the size and arrangement of data, but on the number of times the disk can IO per second. For example, the data of each request such as Web service and Mail service is very small. Random IO will generate more requests per second at the same time, so how many times the disk can IO per second is the key.

Iostat-kx 1avg-cpu:% user% nice% system% iowait% steal% idle 1.75 0.00 0.75 0.25 0.00 97.26

Device: rrqm/s wrqm/s rash s wbank s rkB/s wkB/s avgrq-sz avgqu-sz await svctm% utilsdb 0.00 52.00 0.00 57.00 0.00 436.00 15.30 0.03 0.54 0.23 1.30

Avg-cpu:% user% nice% system% iowait% steal% idle 1.75 0.00 0.75 0.25 0.00 97.24

Device: rrqm/s wrqm/s ramp s wbank s rkB/s wkB/s avgrq-sz avgqu-sz await svctm% utilsdb 0.00 56.44 0.00 66.34 0.00 491.09 14.81 0.04 0.54 0.19 1.29

According to the above formula: 436.00 / 57.00 = 7.65 KB per IO,491.09 / 66.34 = 7.40 KB per IO. Compared with the sequential IO, it is found that the KB per IO of the random IO is so small that it can be ignored. It can be seen that the important thing for the random IO is the number of IOPS per second, not the throughput capacity (KB per IO) of each IO.

4.7 SWAP

The swap device is used when the system does not have enough physical memory to handle all requests, and the swap device can be a file or a disk partition. Be careful, however, that using swap is very expensive. If the system has no physical memory available, it will swapping frequently, and if the data that the swap device and the program are about to access are on the same file system, it will encounter serious IO problems, which will eventually cause the whole system to lag or even crash. The swapping status between swap devices and memory is an important reference for judging Linux system performance. We already have many tools that can be used to monitor swap and swapping conditions, such as top, cat / proc/meminfo, vmstat and so on:

$cat / proc/meminfoMemTotal: 8182776 kBMemFree: 2125476 kBBuffers: 347952 kBCached: 4892024 kBSwapCached: 112 kB...SwapTotal: 4096564 kBSwapFree: 4096424 kB...

$vmstat 1procs-memory- swap---io---- system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 12 260008 2188 1444 11824 2584 12664 2584 1347 1174 140 0 86 0 21 262140 2964 5852 24912 17304 24952 17304 4737 2341 86 10 00

five。 Network

Network monitoring is the most complex of all Linux subsystems, and there are too many factors in it, such as delay, blocking, collision, packet loss and so on. Worse, routers, switches and wireless signals connected to Linux hosts will affect the whole network and it is difficult to judge whether it is because of the problems of Linux network subsystem or other devices, which increases the complexity of monitoring and judgment. All the network cards we use now are called adaptive network cards, which means that they can be automatically adjusted according to the different network speeds and working modes caused by different network devices on the network. We can use the ethtool tool to view the configuration and working mode of the Nic:

# / sbin/ethtool eth0Settings for eth0: Supported ports: [TP] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT / Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: G Wake-on: g Current message level: 0x000000ff Link detected: yes

The example given above shows that there are three options for the network card: 10baseT, Speed 100baseT and 1000baseT, which are currently adapting to 100baseT (Speed: 100Mb/s). You can use the ethtool tool to force the Nic to work under 1000baseT:

# / sbin/ethtool-s eth0 speed 1000 duplex full autoneg off

5.1 iptraf

There are network cables (or wireless), routers, switches and other devices between the two hosts. One way to test the network performance between the two hosts is to send data to each other between the two systems and count the results to see how the throughput, latency and speed are. Iptraf is a good tool for viewing native network throughput. It supports text and graphical interface and is very intuitive. The picture below shows that the transmission rate of this Linux system is a bit slow in a network of 100 mbps, Outgoing rates is only 66 mbps.

# iptraf-d eth0

5.2 netperf

Netperf runs in client/server mode and can test terminal throughput more variously than iptraf. Start netserver on the server side first:

# netserverStarting netserver at port 12865Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC

Then, on the client side, test the server and perform a TCP test that lasts 10 seconds:

# netperf-H 172.16.38.36-l 10TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.38.36 (172.16.38.36) port 0 AF_INETRecv Send SendSocket Socket Message ElapsedSize Size Size Time Throughputbytes bytes bytes secs. 10 ^ 6bits / sec

87380 16384 16384 10.32 93.68

As can be seen from the above output, the network throughput is around 94mbps, which is a good performance for 100mbps's network. The above test is in the case that the server and client are located on the same LAN, and the LAN is wired. You can also try different networks with different structures and rates, for example: there are several routers in the middle of the network, and the client is in wi-fi, VPN and so on.

Netperf can also test the number of TCP requests and responses per second by establishing a TCP connection and sending packets sequentially. The following output shows that the processing rate is 243p when TCP requests uses 2K and responses uses 32K:

# netperf-t TCP_RR-H 172.16.38.36-l 10-r 2048 port REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.38.36 (172.16.38.36) TCP 0 AF_INETLocal / RemoteSocket Size Request Resp. Elapsed Trans.Send Recv Size Size Time Ratebytes Bytes bytes bytes secs. Per sec

16384 87380 2048 32768 10.00 243.0316384 87380

5.3 iperf

Iperf and netperf operate similarly, but also in server/client mode. Start iperf on the server side first:

# iperf-s-D---Server listening on TCP port 5001TCP window size: 85.3 KByte (default)-- -- Running Iperf Server as a daemonThe Iperf daemon process ID: 5695

Then the server is tested on the client side, where the client first connects to the server side (172.16.38.36) and samples the bandwidth of the network between the server and the client every 5 seconds for 30 seconds:

# iperf-c 172.16.38.36-t 30-I 5---Client connecting to 172.16.38.36 TCP port 5001TCP window size: 16.0 KByte (default)-[3] local 172.16.39.100 port 49515 connected with 172.16.38.36 port 5001 [ID] Interval Transfer Bandwidth [3] 0.0-5. 0 sec 58.8 MBytes 98.6 Mbits/sec [ID] Interval Transfer Bandwidth [3] 5.0-10.0 sec 55.0 MBytes 92.3 Mbits/sec [ID] Interval Transfer Bandwidth [3] 10.0-15.0 sec 55.1 MBytes 92.4 Mbits/sec [ID] Interval Transfer Bandwidth [3] 15.0-20.0 sec 55.9 MBytes 93.8 Mbits/sec [ID] Interval Transfer Bandwidth [3] 20.0-25.0 sec 55.4 MBytes 92.9 Mbits/sec [ID] Interval Transfer Bandwidth [3] 25.0-30.0 sec 55.3 MBytes 92.8 Mbits/sec [ID] Interval Transfer Bandwidth [3] 0.0-30.0 sec 335 MBytes 93.7 Mbits/sec

5.4 tcpdump and tcptrace

Tcmdump and tcptrace provide a more detailed analysis method, first using tcpdump to capture packets as required to output the results to a file, and then using tcptrace to analyze the file format. This combination of tools can provide some information that is difficult to find with other tools:

# / usr/sbin/tcpdump-w network.dmptcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes511942 packets captured511942 packets received by filter0 packets dropped by kernel

# tcptrace network.dmp1 arg remaining, starting with 'network.dmp'Ostermann's tcptrace-- version 6.6.7-- Thu Nov 4, 2004

511677 packets seen, 511487 TCP packets tracedelapsed wallclock time: 0 pkts/sec analyzedtrace file elapsed time 00.510291, 1002714 pkts/sec analyzedtrace file elapsed time: 0:02:35.836372TCP connection info: 1: zaber:54581-boulder:111 (A2b) 6 > 5

< (complete) 2: zaber:833 - boulder:32774 (c2d) 6>

five

< (complete) 3: zaber:pcanywherestat - 172.16.39.5:53086 (e2f) 2>

3 257 5

< (complete) 6: 172.16.39.100:47592 - zaber:36814 (k2l) 255380>

255378

< (reset) 7: breakpoint:45510 - zaber:7012 (m2n) 9>

five

< (complete) 8: zaber:35813 - boulder:111 (o2p) 6>

five

< (complete) 9: zaber:837 - boulder:32774 (q2r) 6>

five

< (complete) 10: breakpoint:45511 - zaber:7012 (s2t) 9>

five

< (complete) 11: zaber:59362 - boulder:111 (u2v) 6>

five

< (complete) 12: zaber:841 - boulder:32774 (w2x) 6>

five

< (complete) 13: breakpoint:45512 - zaber:7012 (y2z) 9>

five

< (complete) tcptrace 功能很强大，还可以通过过滤和布尔表达式来找出有问题的连接，比如，找出转播大于100 segments 的连接： # tcptrace -f'rexmit_segs>

100 'network.dmp

If you find that there is a problem with connection # 10, you can view additional information about this connection:

# tcptrace-o10 network.dmp

The following command uses tcptrace's slice mode, and the program automatically creates a slice.dat file in the current directory that contains broadcast information every 15 seconds:

# tcptrace-xslice network.dmp

# cat slice.datdate segs bytes rexsegs rexbytes new active- 16 cat slice.datdate segs bytes rexsegs rexbytes new active- 58 Suzhou 50.244708 85055 4513418 006 616 purl 59: 05.244708 110921 5882896 00 0 216:59:20.244708 126107 6697827 00 1 316:59:35.244708 151719 8043597 00 0 216:59:50.244708 37296 1980557 00 0 317:00:05.244708 67 8828 00 2 317:00:20.244708 149 22053 00 1 217:00:35.244708 30 4080 00 0 117:00:50.244708 39 5688 00 0 117:01:05.244708 67 8828 00 2 317 01 11.081080 37 4121 01 3 and above are all the contents of the article "how to achieve performance Monitoring in Linux" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.