Shulou(Shulou.com)06/01 Report--

This article is to share with you the content of a case of IO monitoring and analysis under Linux. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. System-level IO monitoring

1.1 、 iostat

Iostat-xdm 1 # personal habits

% util represents how busy the disk is. 100% indicates that the disk is busy and 0% indicates that the disk is idle. Note, however, that a busy disk does not mean a high disk (bandwidth) utilization

The size of IO requests submitted by argrq-sz to the driver layer is generally not less than 4K and not greater than max (readahead_kb, max_sectors_kb)

It can be used to judge the current IO mode. In general, especially when the disk is busy, the larger is the order, and the smaller is random.

Svctm the service time of an IO request. For a single disk, when it is read at random, it is basically about 7ms, i.e. seek + rotation delay time.

1.2. The relationship between statistics

% util = (r _ svctm s + w _ r _ s) * queue length / 1000 # queue length = arrival rate * average service time

Avgrq-sz = (rMB/s + wMB/s) * 2048 / (rUniqqs + wUnips) # 2048 is 1m / 512m

1.3. Summary

Iostat counts the IO data submitted directly to the device after the general block layer is merged (rrqm/s, wrqm/s), which can reflect the IO status of the system as a whole, but it has the following two disadvantages:

It is far away from the business layer and does not correspond to the write,read in the code (it is also difficult to correspond to the system pre-read + pagecache + IO scheduling algorithm and other factors)

Is the system level, can not be accurate to the process, for example, can only tell you that the disk is busy, but there is no way to tell you who is busy and what?

2. Process-level IO monitoring

2.1, iotop and pidstat (rhel6u series only)

Iotop, as its name implies, is the io version of top

Pidstat, as the name implies, is the stat of the statistical process (pid), and the stat of the process naturally includes the IO status of the process.

With both commands, you can count IO status by process, so you can know:

1) which processes are taking up IO in the current system, and what is the percentage?

2) is the process that occupies IO reading? Or are you writing? What is the amount of reading and writing?

There are many parameters for pidstat. Here are some more commonly used parameters:

Pidstat-d 1 # shows only IO

Pidstat-u-r-d-t 1

#-d IO information

#-r Page missing and memory information

#-u CPU utilization

#-t count in threads

# 1 count once in 1 second

Iotop, it's simple. Just hit the command.

2.2 、 block_dump, iodump

Both iotop and pidstat rely on statistics exported from the / proc/pid/io file, which is not available for older kernels, such as rhel5u2.

So we have to replace it with the above two commands:

Echo 1 > / proc/sys/vm/block_dump # enable block_dump and enter io information into dmesg at this time

# dmesg

Watch-n 1 "dmesg-c | grep-oP" w + (d +): (WRITE | READ) "| sort | uniq-c"

# non-stop dmesg-c

Echo 0 > / proc/sys/vm/block_dump # close when not in use

2.3. Summary

From the process-level IO monitoring, we can see:

Two questions that cannot be answered by system-level IO Monitoring

Relatively close to the business layer (for example, you can count the number of reads and writes to processes)

But there is no way to connect with the read,write of the business layer, and the granularity is coarse, so there is no way to know which files have been read and written by the current process. Time-consuming? Size?

3. Business-level IO monitoring

3.1 、 ioprofile

The ioprofile command is essentially lsof + strace

Ioprofile can solve three problems:

What files (read, write) have been read and written at the business level during the current process?

What is the number of reads and writes? (read, the number of calls to write)

How much data is read and written? (read, number of byte of write)

3.2. Examples

Suppose a behavior triggers an IO action in the program, for example: "A page click causes the background to read the A _ Magi B ~ J C file"

. / io_event # suppose you simulate IO behavior once, reading A file once, B file 500 times, and C file 500 times

Ioprofile-p `pidof io_ event`-c count # number of reads and writes

Ioprofile-p `pidof io_ event`-c times # time consuming to read and write

Ioprofile-p `pidof io_ event`-c sizes # read and write size

Note: ioprofile only supports multithreaded programs, not single-threaded programs. For IO business-level analysis of single-threaded programs, strace is sufficient.

3.3. Summary

Ioprofile is essentially strace, so you can see the call trajectory of read,write and do io analysis at the business layer.

4. File-level IO monitoring

File-level IO monitoring can complement / complement "business-level and process-level" IO analysis

File-level IO analysis, mainly for a single file, to see which processes are currently reading and writing to a file.

Lsof or ls / proc/pid/fd

Inodewatch.stp

4.1 、 lsof

Check which processes open the current file

The lsof.. / io # io directory is currently opened by two processes, bash and lsof

Lsof commands can only view static information, and "open" does not necessarily "read". For commands such as cat and echo, opening and reading are instantaneous, and lsof is difficult to capture.

4.2 、 inodewatch.stp

Inodewatch.stp can be used to make up for it.

Stap inodewatch.stp major minor inode # Primary device number, Auxiliary equipment number, File inode Node number stap inodewatch.stp 0xfd 0x00 523170 # Primary device number, Auxiliary equipment number, inode number, which can be obtained through the stat command

Thank you for reading! This is the end of this article on "the case of IO monitoring and analysis under Linux". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.