Shulou(Shulou.com)06/01 Report--

Disk busy when appending AOF

Screenshot of redis error log

The following figure shows the AOF workflow flow chart.

AOF work flow chart

1. All write commands append aof_buf (in buffer).

The 2.AOF buffer synchronizes to the hard disk according to the corresponding policy.

3. As the AOF file becomes larger and larger, it is necessary to rewrite the AOF file regularly to achieve the purpose of compression (there will be 100 records for a key and a new 100 times, and only the last entry will be left after rewriting).

4. When the redis server is restarted, the AOF file can be loaded for data recovery.

Append aof blocking to the original master database

The number of blocking times appended to the original primary database of ip addresses.

Aof_delayed_fsync:154

Flowchart of using everysec (per second) flushing strategy

1. The main thread is responsible for the AOF buffer

The 2.AOF thread is responsible for synchronizing disk operations per second and recording the last synchronization time.

3. The main thread compares the AOF synchronization time:

3.1 if the last synchronization time is within two seconds, the main thread returns directly.

3.2 if the time since the last synchronization is more than two seconds (conscious that it is still synchronizing), the main thread will be blocked until the synchronization is complete.

Statistics on the blocking times of other main databases in the whole cluster

The AOF of 15 main libraries was busy because the disk was busy during SYNC.

The number of blocking when appending AOF, of which 11 main libraries have been blocked

The number of aof_delayed_fsync blocking.

Ip1

# Persistence

Aof_delayed_fsync:1

Ip2

# Persistence

Aof_delayed_fsync:18

Ip3

# Persistence

Aof_delayed_fsync:0

Ip4

# Persistence

Aof_delayed_fsync:1

Solution

1. The no-appendfsync-on-rewrite parameter is turned on and off by default, which means that no sync operation is performed during AOF rewriting, which does not fundamentally solve the problem, because AOF rewriting does not occur before the failover.

two。 Shut down AOF. If a group (master-slave) goes down at the same time, the data will be lost for 5 minutes. If no AOF file is found when starting redis, redis will select RDB to recover the data. The frequency of rdb copy-on-write to disk is 5 minutes.

3. Increase disk write speed.

Restart loading flow chart

RDB materialization to disk frequency diagram

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.