Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to solve the Linux iowait problem. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What are you waiting for?

The IO wait is related to the CPU resource on the server.

The iowait column on the top command output shows the percentage of time that the processor has waited for Imax O to complete. It indicates that the system is waiting for disk or network IO. Because the system is waiting for these resources, CPU cannot be fully utilized.

Learn more about what I'm waiting for here.

Ihop O waiting is a problem that requires advanced tools for debugging, and of course, the basics have many advanced uses.

Determine if the Icano problem is causing the system to slow down

We can use multiple commands to verify that the system is slowing down due to Iramp O, but the simplest is the UNIX command top.

[[email protected] ~] # toptop-15:19:26 up 6:10, 4 users, load average: 0.00,0.01,147total, 1 running, 146sleeping, 0 stopped, 0 zombie%Cpu (s): 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 96.0 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem: 999936 total, 121588 free, 328672 used 549676 buff/cacheKiB Swap: 2097148 total, 2095792 free, 1356 used. 450460 avail Mem

From the CPU line, we can see the percentage of CPU wasted on iPlus waiting; the higher this number is, the more CPU resources are waiting for iPlus permission.

Wa-iowait AmountoftimetheCPUhasbeenwaitingfor iowait AmountoftimetheCPUhasbeenwaitingfor O to complete.

Learn about CPU usage in Linux Top

Find the disk being written

The above top command explains the I _ Iostat O wait as a whole, but does not indicate which disk is affected. To know which disk is causing the problem, we use another command, the wait command. For more information, see this article on troubleshooting disk problems in Linux.

[[email protected] ~] # iostat-x 2 5Linux 3.10.0-514.el7.x86_64 (localhost.localdomain) 03 / 03 / 2017 _ x86 / 64 _ (1 CPU) avg-cpu:% user% nice% system% iowait% steal% idle 0.34 0.00 0.31 0.01 0.00 99.33Device: rrqm/s wrqm/s rmags WUnix rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm% utilsda 0.00 0.05 1.16 0.17 39.00 17.38 84.60 0.00 2.17 0.87 11.14 0.65 111.41scd0 0.00 0.00 0.00. 00 8.00 0.00 0.64 0.64 0.00 0.64 0.00dm-0 0.00 0.00 1.10 0.20 37.85 17.21 84.71 0.00 2.43 0.90 10.88 0.66 0.09dm-1 0.00 0.00 0.01 0.02 0. 07 0.08 9.70 0.00 1.42 0.27 2.05 0.09 0.00

In the above example, Iostat will update every 2 seconds, print the information five times, and the option for-X is to print extended information.

The first Iostat report prints statistics after the system's last boot, which means that in most cases, the first print should be ignored, and the rest of the report is based on the last interval. For example, this command will print five times, the second report is the statistics of the first report, the third is based on the statistics of the second report, and so on

In the above example, the utilization of SDA is 111.41%, which is a good indication that the process is being written to SDA disk.

In addition to the% utilization, we can get richer resources from iostat, such as read / write requests per millisecond (rrqm/s&wrqm/s), reads and writes per second (r/s&w/s), and more. In the above example, our project seems to be reading and writing a lot of information. This is very useful for us to find the right process.

Find the process that caused the high Imax O to wait

[[email protected] ~] # iotopTotal DISK READ: 0.00B email protected s | Total DISK WRITE: 0.00 B/sActual DISK READ: 0.00B 1028COMMAND | Actual DISK WRITE: 0.00B TID PRIO USER DISK READ DISK WRITE SWAPIN IO > COMMAND 1028 be/4 root 0.00B 1028COMMAND 0.00% sshd [[email protected] ~] # lsof-p 1028COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEsshd 1028 root cwd DIR 253,0 233 64/ sshd 1028 root rtd DIR 253,0 233 64/ sshd 1028 root txt REG 253,0 819640 2393730 / usr/sbin/sshdsshd 1028 root mem REG 253,0 61752 180464 / usr/lib64/libnss_files-2.17.sosshd 1028 root mem REG 253,0 43928 180476 / usr/lib64/librt-2.17.sosshd 1028 root mem REG 253,0 15688 269136 / usr/lib64/ Libkeyutils.so.1.5sshd 1028 root mem REG 253,0 62744 482870 / usr/lib64/libkrb5support.so.0.1sshd 1028 root mem REG 253,0 11384 180425 / usr/lib64/libfreebl3.sosshd 1028 root mem REG 253,0 143352 180472 / usr/lib64/libpthread-2.17.sosshd 1028 root mem REG 253,0 251784 202440 / usr/lib64/libnspr4.sosshd 1028 root mem REG 253,0 20016 202441 / usr/lib64/libplc4.sosshd 1028 root mem REG 253,0 15768 202442 / usr/lib64/ Libplds4.sosshd 1028 root mem REG 253,0 182056 202443 / usr/lib64/libnssutil3.sosshd 1028 root mem REG 253,0 1220240 650074 / usr/lib64/libnss3.sosshd 1028 root mem REG 253,0 164048 650076 / usr/lib64/libsmime3.sosshd 1028 root mem REG 253,0 276752 650077 / usr/lib64/libssl3.sosshd 1028 root mem REG 253,0 121296 269112 / usr/lib64/libsasl2.so.3.0.0sshd 1028 root mem REG 253,0 398264 202404 / usr/lib64/libpcre.so.1.2 .0sshd 1028 root mem REG 253,0 2116736 180446 / usr/lib64/libc-2.17.sosshd 1028 root mem REG 253,0 15848 202439 / usr/lib64/libcom_err.so.2.1sshd 1028 root mem REG 253,0 202568 482862 / usr/lib64/libk5crypto.so.3.1sshd 1028 root mem REG 253,0 959008 482868 / usr/lib64/libkrb5.so.3.3sshd 1028 root mem REG 253,0 324888 482858 / usr/lib64/libgssapi_krb5.so.2.2sshd 1028 root mem REG 253,0 110632 180474 / usr/lib64/libresolv-2.17.sosshd 1028 root mem REG 253,0 40640 180450 / usr/lib64/libcrypt-2.17.sosshd 1028 root mem REG 253,0 113152 180456 / usr/lib64/libnsl-2.17.sosshd 1028 root mem REG 253,0 90664 202424 / usr/lib64/libz.so.1.2.7sshd 1028 root mem REG 253,0 14432 186432 / usr/lib64/libutil-2.17.sosshd 1028 root mem REG 253,0 61872 766946 / usr/ Lib64/liblber-2.4.so.2.10.3sshd 1028 root mem REG 253,0 344280 766948 / usr/lib64/libldap-2.4.so.2.10.3sshd 1028 root mem REG 253,0 19344 180452 / usr/lib64/libdl-2.17.sosshd 1028 root mem REG 253,0 2025472 482880 / usr/lib64/libcrypto.so.1.0.1esshd 1028 root mem REG 253,0 23968 202508 / usr/lib64/libcap-ng.so.0.0.0sshd 1028 root mem REG 253,0 155744 202421 / usr/lib64/libselinux.so.1sshd 1028 root mem REG 253,0 61672 539049 / usr/lib64/libpam.so.0.83.1sshd 1028 root mem REG 253,0 122936 202512 / usr/lib64/libaudit.so.1.0.0sshd 1028 root mem REG 253,0 42520 298848 / usr/lib64/libwrap.so.0.7.6sshd 1028 root mem REG 253,0 11328 568388 / usr/lib64/libfipscheck.so.1.2.1sshd 1028 root mem REG 253,0 155064 180439 / usr/lib64/ld-2.17.sosshd 1028 root 0u CHR 1 CHR 3 0t0 5930 / dev/nullsshd 1028 root 1u CHR 1 0t0 5930 / dev/nullsshd 1028 root 2u CHR 1 0t0 5930 / dev/nullsshd 1028 root 3u IPv4 21185 0t0 TCP *: ssh (LISTEN) sshd 1028 root 4u IPv6 21194 0t0 TCP *: ssh (LISTEN)

To further confirm that these files are read and written frequently, we can view them using the following command

[email protected] ~] # df / tmpFile system 1K-block used available used% mount point/ dev / mapper / cl-root 17811456 3981928 13829528 23% /

Based on the results of the above command, we can determine that / tmp is the root of the logical disk of our environment

[[email protected] ~] # pvdisplay-Physical volume-PV Name / dev/sda2 VG Name cl PV Size 19.00 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 4863 Free PE 0 Allocated PE 4863 PV UUID 4QfaOy-DNSO-niK1-ayn2-K6AY-WZMy-9Nd2It, this is how to solve the Linux iowait problem shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.