Shulou(Shulou.com)06/03 Report--

This article mainly explains "analyzing an atypical Redis blocking". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "analyze an atypical Redis blocking".

The south is gradually entering the best time of the year, and users are starting to stir up. After taking a look at the data, active users have already double very far, and will soon triple.

One bleary-eyed morning, when I was watching the data silently yy, the online alarm began. MMP, the user uproar in the morning is growing faster than expected. The colleague immediately opened the stereoscopic monitor and glanced at it. Combined with the error log of the service, he quickly locked the problem to a Redis instance (in fact, since the stereoscopic monitor was launched, the processing flow basically changed from the previous

Has become

). The team is pretty efficient, and the reason is AOF persistence:

This is the Redis configuration at that time:

127.0.0.1 6379 > config get * append*

1) "no-appendfsync-on-rewrite"

2) "no"

3) "appendonly"

4) "yes"

5) "appendfsync"

6) "everysec"

From a configuration point of view, the reason is theoretically clear: our Redis example uses AOF for persistence (appendonly), and the appendfsync strategy uses everysec flushing. But AOF over time, the file will become larger and larger, so Redis also has a rewrite strategy to achieve AOF file weight loss, but the result is idempotent. Our no-appendfsync-on-rewrite strategy is no. This causes the appendfsync to be blocked during rewrite operations. If the current AOF file is large, the corresponding rewrite time will be longer and the appendfsync will be blocked for longer.

This is not a new problem, which is encountered in many business scenarios that open AOF. There are several solutions:

Set no-appendfsync-on-rewrite to yes. This avoids contention with appendfsync for file handles, but there is a risk that AOF will be lost during rewrite.

Add a slave node to the current Redis instance, the current node is set to master, and then the master node closes the AOF,slave node and enables AOF. The risk of this approach is that if the master dies, data that has not yet been synchronized to the salve will be lost.

We took a compromise: set no-appendfsync-on-rewrite to yes in the master node setting and add the slave node.

Theoretically, the problem should be solved, right? Ah, in theory, indeed.

On the first day after the revision, the problem appeared again. Are you surprised? are you surprised?

As a result, the partner reviewed the Redis log when something went wrong:

There are two comparisons:

The first few AOF log alarm logs occurred between 3pm and 5pm, when our overall system load was very low.

The early morning alarm log is not an alarm of a Redis instance, but an alarm of all Redis instances on the machine.

Thank you for your reading. the above is the content of "analyzing an atypical Redis blocking". After the study of this article, I believe you have a deeper understanding of the problem of analyzing an atypical Redis blocking, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.