Shulou(Shulou.com)06/01 Report--

The editor will share with you how to ensure the high concurrency of Redis. I hope you will gain a lot after reading this article. Let's discuss it together.

It is almost impossible for a stand-alone redis to say that the QPS exceeds 100000 +, usually in tens of thousands.

Unless there are some special circumstances, such as your machine performance is very good, the configuration is very high, the physical machine, maintenance is particularly good, and your overall operation is not too complicated.

Through the master-slave architecture, Redis realizes the separation of read and write, the master node is responsible for writing, and synchronizes the data to other slave nodes, and the slave node is responsible for reading, so as to achieve high concurrency.

While Redis has high concurrency, it also needs to accommodate a large amount of data: one master and more slaves, each instance contains complete data, such as redis master 10g of memory, in fact, you can only accommodate 10g of data. If your cache contains a large amount of data, up to tens of grams, or even hundreds of grams, or even a few tons, then you need a redis cluster, and with redis clustering, you can provide read and write concurrency that may be hundreds of thousands per second.

The core mechanism of replication

Redis replicates data to slave nodes asynchronously, but starting with redis 2.8, slave node periodically confirms the amount of data it replicates.

One master node can be configured with multiple slave node.

Slave node can also connect to other slave node

When slave node makes a copy, block master node will not work properly.

When slave node does replication, it will not block its own query operation, it will use the old dataset to provide services; but when the replication is completed, it needs to delete the old dataset and load the new dataset, and then the external service will be suspended.

Slave node is mainly used for horizontal expansion and separation of read and write. The expanded slave node can improve the read throughput.

The significance of master persistence to the Security of Master-Slave Architecture

If the master-slave architecture is adopted, it is recommended that master node persistence be enabled!

It is not recommended to use slave node as the data hot backup of master node, because in that case, if you turn off the persistence of master, the data may be empty when master is down and restart, and then the salve node data may be lost as soon as it is replicated.

Second, the various backup schemes of master, should we do it, in case all the local files are lost; choose a rdb from the backup to restore master; so as to ensure that there is data when master starts.

The process of synchronizing data to slave by master

When a slave node is started, it sends a PSYNC command to master node

If this is slave node reconnecting to master node, then master node will only copy the missing data to slave; otherwise, if slave node connects to master node for the first time, then full resynchronization will be triggered

When you start full resynchronization, master starts a background thread, starts to generate a RDB snapshot file, and caches all write commands received from the client in memory.

After the RDB file is generated, master sends the RDB to slave,slave, which is written to the local disk and then loaded into memory from the local disk. Master then sends in-memory cached write commands to slave,slave and synchronizes the data.

If slave node has a network failure with master node and is disconnected, it will automatically reconnect. If master finds that multiple slave node are reconnecting, it simply initiates a rdb save operation that serves all slave node with one piece of data.

Continuation of breakpoint of master-slave replication

Since redis 2.8, the breakpoint continuation of master-slave replication has been supported. If the network connection is disconnected during the master-slave replication process, then you can continue to copy where you copied last time, instead of copying a copy from scratch.

Master node will be common in memory. Both backlog,master and slave will save a replica offset and a master id,offset will be saved in backlog.

If the network connection between master and slave is disconnected, slave will let master continue to copy from the last replica offset

But if no corresponding offset is found, then a resynchronization will be executed

Diskless replication

Master creates a rdb directly in memory and then sends it to slave. It will not land on its own local disk.

Repl-diskless-sync

Repl-diskless-sync-delay, wait a certain amount of time before starting replication, because you have to wait for more slave to reconnect.

Expired key processing

Slave does not expire key, just waits for master to expire key.

If master expires a key or eliminates a key through LRU, a del command is simulated and sent to slave.

Complete process of replication

Slave node starts, saving only master node information, including master node's host and ip (configured by slaveof in redis.conf), but the replication process has not started

There is a scheduled task within slave node to check every second whether there is a new master node to connect to and replicate. If found, establish a socket network connection with master node.

Slave node sends ping commands to master node

Password authentication. If master sets requirepass, then salve node must send masterauth password to authenticate.

Master node performs a full copy for the first time, sending all data to slave node

Master node continues to write commands and copy them asynchronously to slave node.

Core mechanisms related to data synchronization

Refers to the full copy performed when slave connects to msater for the first time, and the mechanism for some of your details in that process.

(1) both master and slave maintain an offset

Master will continue to accumulate in itself, offset,slave will continue to accumulate offset in itself.

Slave reports its own offset to master every second, and master saves the offset of each slave.

This is not to say that it is specifically used for full replication. It is mainly that both master and slave need to know the offset of their respective data in order to know the data inconsistency between each other.

(2) backlog

Master node has a backlog, which defaults to 1MB size

When master node copies data to slave node, it will also write a copy of the data synchronously in backlog.

Backlog is mainly used to do incremental replication with full replication interruption.

(3) master run id

Info server, you can see master run id

If you locate master node according to host+ip, it is unreliable. If master node restarts or the data changes, then slave node should be distinguished according to different run id, and full replication should be done if run id is different.

If you need to restart redis without changing run id, you can use the redis-cli debug reload command

(4) psync

Copy from the master node using psync from the slave node, psync runid offset

Master node will return response information according to its own situation. It may be that FULLRESYNC runid offset triggers full replication, or CONTINUE triggers incremental replication.

Full copy

Master executes bgsave to generate a rdb snapshot file locally

Master node sends the rdb snapshot file to salve node. If the rdb replication time exceeds 60 seconds (repl-timeout), then slave node will think that the replication failed, and you can adjust this parameter appropriately.

For machines with gigabit network cards, 100MB of 6G files are generally transferred per second, which is likely to exceed 60s.

When master node generates rdb, it caches all new write commands in memory. After salve node saves the rdb, it copies the new write commands to salve node.

Client-output-buffer-limit slave 256MB 64MB 60, if the memory buffer consumption continuously exceeds 64MB during replication, or exceeds 256MB at one time, then replication stops and replication fails.

After receiving the rdb, slave node empties its old data, reloads rdb into its own memory, and provides services based on the old data version

If AOF is enabled in slave node, BGREWRITEAOF will be executed immediately, rewriting AOF

Incremental replication

If the master-slave network connection is disconnected during full replication, incremental replication will be triggered when salve reconnects to master

Master gets part of the lost data directly from its backlog and sends it to slave node. The default backlog is 1MB.

Msater obtains data from backlog according to the offset in psync sent by slave.

Heartbeat

Master and slave nodes send heartbeat messages to each other.

By default, master sends every 10 seconds. Heartbeat,salve node sends a heartbeat every 1 second.

Asynchronous replication

Every time master receives a write command, it now writes data internally and then asynchronously sends it to slave node

After reading this article, I believe you have a certain understanding of how to ensure the high concurrency of Redis. Want to know more about it. Welcome to follow the industry information channel. Thank you for your reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.