Shulou(Shulou.com)06/01 Report--

This paper mainly records a simple, typical failure, the cause of the problem is very simple, the problem is also very simple, students must pay attention to, a careless will have an impact on the main database.

Welcome to reprint, please indicate the author and source.

Author: Zhang Zheng blog: http://space.itpub.net/26355921 QQ:176036317 if you have any questions, please feel free to contact us.

Summary of the problem: a week ago, a mysql server had a hardware failure and shut down. We submitted an application to the students who are in charge of this area, and they are responsible for repairing the server. After the server was repaired today, they booted it up. The four mysql instances on the server start automatically after booting and start pulling the binlog of the main library. Due to the long downtime of this server, more logs are lost, and the binlog of the main library is pulled, which leads to problems in the network of the main library.

Phenomenon:

First of all, we didn't realize that it was caused by a broken server restarting the main library binlog, because we had no idea what was going on with this server, only that we had a server repaired a week ago. We have no idea about the specific situation, whether it has been repaired or not, and whether it has been turned on or not. Under such circumstances, I suddenly heard a classmate on the network say that there was a machine in mysql with too much network traffic, which made the business feel very slow and lasted for a total of 17 minutes. As a matter of fact, there is not much clue to this.

Troubleshooting:

Looking at processlist, full log, and slow log, there is no problem.

Check the monitoring and found that the read IO of the server increased suddenly during that period. By looking at the history of processlist, it is found that for a period of time, the user status of master-slave replication is waiting for net, and through its IP, it is found that the server is a slave server that broke down a week ago.

Conclusion: there are 4 instances on this server. after the server starts, the mysql instance starts automatically and starts to pull binlog to the main database. The daily binlog quantity of each main database is about 6G, and the binlog of 4 instances in a week is about 160g.

Question: 1, when to repair the broken server, when to boot, we can not control, also do not know, and do not pay attention to 2, this case is actually very simple, very typical may cause impact or failure of the case, we are not alert to this phenomenon in advance, although we know that this is a very easy problem, but in our case, there is no awareness of this. Therefore, it leads to the occurrence of the event. 3. There is a lack of effective monitoring of network traffic.

Solution: 1, all servers, cancel the boot to start mysql automatically, after the server is powered on, start the instance artificially and stop slave. (in this way, if there are many servers, it may be too troublesome, so it is better to record it this way than to have an impact.) 2. Be aware of the problem and include it in the common sense base or work manual to avoid the problem.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.