Shulou(Shulou.com)06/02 Report--

As soon as I went to work this morning, some colleagues reported that several websites of the company could not be opened and logged into the database.

The server (windows) is found to be very stuck, so restart the server. After entering the system, the problem remains after a while. Check the system process and find that the mysql occupancy rate has reached 99%. It is certain that there is a problem with the mysql connection:

Netstat-an

192.168.12.13:3306 192.168.12.12:30443 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30444 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30445 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30446 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30447 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30448 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30449 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30450 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30451 TIME_WAIT

192.168.12.13:3306 192.168.12.12:30452 TIME_WAIT

......

According to the 3-way handshake disconnection rule defined by the TCP protocol, the socket that initiates the active shutdown of the socket will enter the TIME_WAIT state, and the TIME_WAIT state will last for 2 MSL (Max Segment Lifetime). Under Windows, the default is 4 minutes, that is, 240s. The socket in the TIME_WAIT state cannot be recycled. The specific phenomenon is that for a server that handles a large number of short connections, if the server actively closes the client connection, it will lead to a large number of socket in the TIME_WAIT state on the server side, even much more than the socket in the Established state, which will seriously affect the processing capacity of the server, and even exhaust the available socket and stop the service. TIME_WAIT is a mechanism used by TCP protocol to ensure that the redistributed socket will not be affected by the previous delayed retransmission packets, and it is a necessary logical guarantee.

In HKEY_LOCAL_MACHINE\ SYSTEM\ CurrentControlSet\ Services\ Tcpip\ Parameters, add a

DWORD key, set to 60 to shorten the waiting time of TIME_WAIT

Log in to the web server (linux):

Netstat-ae | grep mysql

Tcp 0 0 aaaa:53045 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53044 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53051 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53050 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53049 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53048 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53055 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53054 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53053 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53052 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53059 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53058 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53057 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53056 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53063 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53062 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53061 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53060 192.168.12.3:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53067 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53066 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53065 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53064 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa53071 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53070 192.168.12.13:mysql TIME_WAIT root 0

Tcp 0 0 aaaa:53069 192.168.12.13:mysql TIME_WAIT root 0

It is found that there are a large number of connections in TIME_WAIT state in the system, which can be solved by adjusting kernel parameters.

Vi / etc/sysctl.conf

Edit the file and add the following:

Net.ipv4.tcp_syncookies = 1

Net.ipv4.tcp_tw_reuse = 1

Net.ipv4.tcp_tw_recycle = 1

Net.ipv4.tcp_fin_timeout = 30

Then execute / sbin/sysctl-p to make the parameter take effect.

Net.ipv4.tcp_syncookies = 1 means that SYN Cookies is enabled. When a SYN waiting queue overflow occurs, enable cookies to handle it to prevent a small amount of SYN***, from defaulting to 0, which means that it is turned off.

Net.ipv4.tcp_tw_reuse = 1 means reuse is turned on. Allow TIME-WAIT sockets to be reused for new TCP connections. Default is 0, which means off.

Net.ipv4.tcp_tw_recycle = 1 means to enable fast recycling of TIME-WAIT sockets in TCP connections. Default is 0, which means disabled.

Net.ipv4.tcp_fin_timeout modifies the system's default TIMEOUT time

After modification, re-use

Netstat-ae | grep mysql

Tcp 0 0 aaaa:50408 192.168.12.13:mysql ESTABLISHED nobody 3224651

Tcp 0 0 aaaa:50417 192.168.12.13:mysql ESTABLISHED nobody 3224673

Tcp 0 0 aaaa:50419 192.168.12.13:mysql ESTABLISHED nobody 3224675

Found that a large number of TIME_WAIT no longer exist, the occupancy rate of the mysql process quickly dropped, the website access is normal!

The above is only a temporary solution. Finally, a careful inspection found that it was a system that was newly launched the day before yesterday. Mysql.colse () was not used in the program code, which led to a large number of mysql TIME_WAIT.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.