Shulou(Shulou.com)06/02 Report--

In this issue, Xiaobian will bring you the troubleshooting process on how to analyze disk IO high. The article is rich in content and analyzed and described from a professional perspective. After reading this article, I hope you can gain something.

We set up cacti to monitor the server. Once we checked the disk IO image, we found that the disk IO spiked very high at 3:20-3:35 every morning, and then we wanted to know which process occupied so much IO. Here is the solution:

Write a detection script check_io_process.sh to detect the number of disk IO read and write processes when disk IO occupancy is high:

#!/ bin/bash

# Date: 2013/8/20

# Author: zhangkai

# Description: This script is used to check IO higher process.

# History:

iostat_log=/data/logs/iostat/iostat.log

dmesg_log=/data/logs/iostat/dmesg.log

dstat_log=/data/logs/iostat/dstat.log

if [ ! -d /data/logs/iostat ];then

mkdir -p /data/logs/iostat

fi

add(){

str=$@

sum=`echo ${str// /+}|bc -l`

}

iostat -x 1 5 > $iostat_log

idle_percent=`cat $iostat_log | awk 'BEGIN{flag=0} {if(flag ==1){print $12; flag=0;} if (index($0, "%util" )) {flag = 1;}}'`

add $idle_percent

#Find the average of 5 queries IO occupancy

avg=`echo $sum/5|bc`

if [[ $avg -ge 70 ]];then

echo 1 > /proc/sys/vm/block_dump

echo "-----------------------------------------------" >> $dmesg_log

echo `date "+%Y-%m-%d %H:%M:%S"` >> $dmesg_log

python /data/dmesg_io.py >> $dmesg_log

echo "-----------------------------------------------" >> $dstat_log

echo `date "+%Y-%m-%d %H:%M:%S"` >> $dstat_log

dstat -d --top-bio 1 10 >> $dstat_log

echo 0 > /proc/sys/vm/block_dump

fi

The shell script calls a python script that detects the number of disk IO read and write processes. Here is the code for dmesg_io.py:

#!/ usr/bin/python

# Monitoring per-process disk I/O activity

# written by http://www.vpsee.com

import sys, os, time, signal, re

class DiskIO:

def __init__(self, pname=None, pid=None, reads=0, writes=0):

self.pname = pname

self.pid = pid

self.reads = 0

self.writes = 0

def main():

argc = len(sys.argv)

if argc != 1:

print "usage: ./ iotop"

sys.exit(0)

if os.getuid() != 0:

print "must be run as root"

sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

os.system('echo 1 > /proc/sys/vm/block_dump')

print "TASK PID READ WRITE"

# while True:

os.system('dmesg -c > /tmp/diskio.log')

l = []

f = open('/tmp/diskio.log', 'r')

line = f.readline()

while line:

m = re.match(\

'^(\S+)\((\d+)\): (READ|WRITE) block (\d+) on (\S+)', line)

if m != None:

if not l:

l.append(DiskIO(m.group(1), m.group(2)))

line = f.readline()

continue

found = False

for item in l:

if item.pid == m.group(2):

found = True

if m.group(3) == "READ":

item.reads = item.reads + 1

elif m.group(3) == "WRITE":

item.writes = item.writes + 1

if not found:

l.append(DiskIO(m.group(1), m.group(2)))

line = f.readline()

time.sleep(1)

for item in l:

print "%-10s s d d" % \

(item.pname, item.pid, item.reads, item.writes)

def signal_handler(signal, frame):

os.system('echo 0 > /proc/sys/vm/block_dump')

sys.exit(0)

if __name__=="__main__":

main()

The logs found at 3:20-3:35 are as follows (only some are listed):

[root@localhost iostat]# cat dmesg.log

-----------------------------------------------

2013-08-22 03:23:06

TASK PID READ WRITE

updatedb 18661 2951 0

kjournald 804 0 525

kjournald 1826 0 576

-----------------------------------------------

2013-08-22 03:24:05

TASK PID READ WRITE

updatedb 18661 3007 0

kjournald 804 0 238

kjournald 1826 0 112

flush-8:0 11687 0 18

-----------------------------------------------

2013-08-22 03:25:05

TASK PID READ WRITE

updatedb 18661 2689 0

kjournald 804 0 229

kjournald 1826 0 44

Description is updatedb This process is causing trouble, google check it, this is automatically run by [cron] update system data script.

Its role is to index files in your system so that query commands such as locate and whereis can be executed quickly.

Our server/data/directory generates a large number of small files every day, resulting in high disk IO consumption when building indexes.

The server indexes files on the hard disk regularly every day. Simply put, it establishes a database and stores all file directory information into this database. When using the whereis and locate commands to search for files, it directly reads data from this database. Instead of looking for files on the hard drive like find. Whereas searching for a file takes almost seconds, finding takes minutes or more. updatedb.Conf makes searching many times more efficient. But there are shortcomings, it needs to update the index every day, which will lead to IO load is too high, because it is not updated from time to time, so there will be a search to delete files, can not find new files added, usually we rarely use management. If the number of files is large and the updates are trivial, we can turn this feature off.

Optimization method:

1. Stop indexing the/data directory

vim /etc/updatedb.conf

Find PRUNEPATHS and add the directories you don't want this updatedb indexed to

2. Set the frequency of scheduled updates to decrease from once a day to once a week with the following command:

mv /etc/cron.daily/mlocate.cron /etc/cron.weekly/

Of course, if your server does not need to create an index, you can also remove it directly.

The above is how to analyze the disk IO high problem troubleshooting process shared by Xiaobian for everyone. If there is a similar doubt, please refer to the above analysis for understanding. If you want to know more about it, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.