Shulou(Shulou.com)05/31 Report--

This article shares with you the content of a sample analysis of Redis timeout troubleshooting. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

During my work two days ago, I suddenly received an alarm that Redis was down, and a large group of people were also saying that so-and-so's Redis connection timed out. At first I thought there was a big problem, but who knew it would recover after a while. At that time, I logged on to the server and checked the monitoring. Check out QPS for the first time:

You can see that the QPS is not high, but what happened when you didn't get the data for a while? Then continue to look at the cpu usage of Redis:

You can see that cpu is saturated, which explains why the graph is broken, because redis is single-threaded, and after using cpu 100%, other commands cannot be processed, and zabbix cannot execute the info command to fetch qps. So it is known that the problem is caused by cpu usage saturation, so what is the reason? Then continue to check to see if there are any slow logs during the period when the cpu usage is high:

It does not seem to be the killer who caused the high cpu, which is difficult to troubleshoot. This example is one master and one slave. So let me take a look at the cpu usage of the slave library:

Shit, what's the matter? from the library that is not used, how come the cpu is also used to 74%? It's not scientific, huh? Whatever, see if there is a slow log from the library:

Shit, what's going on? No one uses it from the library. See if it is read-only:

127.0.1 6103 > CONFIG GET "slave-read-only"

1) "slave-read-only"

2) "yes"

127.0.0.1purl 6103 >

It seems to be read-only, which confuses me. Finally, it occurred to me that the master library has big key expiration at this point, and the slow key operation of the master library expiration will not record the slow log, and the key expiration of the slave library is deleted by the DEL instruction initiated by the master library. At this time, slow logs will be recorded from the library. From the above slow logs, you can see the 335ms with the largest DEL operations. No wonder there are application connection timeouts.

Then use the command info commandstats to see:

Thank you for reading! This is the end of this article on "sample Analysis of Redis timeout troubleshooting". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.