Shulou(Shulou.com)06/01 Report--

Recently, I have frequent contact with redis. I found that some things are often used at work, and it took me some time to consolidate. This article is mainly sorted out in a summary way, because the theme of redis is so large that any technical point is the volume of several articles. It can also be said that this article is an overview.

1.redis basic data structure and short structure Compression

Understanding the data structure of redis is helpful to understand the advantages and disadvantages of each data structure and facilitate the design of a reasonable cache structure.

1.1.redis provides five data structures

1.STRING: you can store strings, floating-point types, and integers. String operations can be performed for strings, and addition and subtraction can be performed for floating-point types and integers. Redis will identify its specific type.

2.LIST: linked list, each NODE in the linked list contains a string. You can push in and out of the linked list at both ends.

3.SET: unordered set, there will not be the same set elements, set intersection, union, subtraction operation.

4.HASH: key-value pairs do not need hashing, add, delete, and get key values.

5.ZSET: an ordered collection sorted by a floating-point score.

These data types are quite obvious to be used in scenarios. When we encounter complex cache scenarios, we need to combine these data structures to design.

For example, in the shopping cart scenario, we first need two HASH to store, the first HASH is the relationship between the user and the shopping cart, and the second HASH is the list of items in the shopping cart.

First get the shoppingCartId through userId, and then get the ProductIds of the user's shopping cart through shoppingCartId. Then the product information of the user's shopping cart can be read asynchronously through ProductIds.

Getting shoppingCartId through userId, and then getting ProductIds through shoppingCartId, seems like two steps that can be performed in one redis command. (with the help of redis's pipeline pipeline. )

Figure 1:

When designing cache, we can consider these structures slightly. If there is no ideal structure to expand, we can mix and match these structures to meet the complex cache structure. There is a balance point is to grasp the expiration time, there is no way to set the expiration time of specific item, so specific data objects still need to set expire to ensure the freshness of the data, for example, the STRING:p100 products here.

1.2. Integer set coding of compressed tables and sets

Redis provides a compressed storage method for lists, collections, hashes and ordered collections. Using the short structure provided by redis when there is not much data, you can get a great storage compression ratio.

The short structure implementation corresponding to the data type of redis: list, zset, hash, these three will use ziplist,set to use set integer encoding.

The default storage structure is an entry node linked list to store data. However, entry node is a structured storage. Take list, for example, before and after the entry node contains a pointer to form a two-way linked list, with the contained data in the middle. The data node consists of three parts, the space used by the current string, the space left by the current string, and the value of the current string.

These loose structures bring some storage overhead. Redis provides a ziplist (compressed table) function. Once the compressed table is turned on, the storage structure originally used in entry node will be stored using serialized byte sequences. This has both advantages and disadvantages, the advantage is less storage space, the disadvantage is inflexibility. Less storage space not only brings high space utilization, but also a big advantage is that it is beneficial to execute master\ slave repliaction or BGSAVE, and the storage or bandwidth consumption is small.

Hash is used behind set to ensure that there are no duplicates. When using short structure storage for set, redis will use integer set coding for storage.

Also, if we set the maximum compression limit after the redis will restore the entry node linked list is stored in the way. Because, if you have too much data in the compressed table, reading or modifying it will affect performance. You may need to decode, encode the entire compressed list, and so on.

2.redis transaction

Redis provides transaction functionality, and you can use transactions to handle sensitive scenarios where data consistency is required. There is no such thing as an isolation level for redis transactions, but performance is what it pursues. All the command contained in the transaction is executed one after another, and all requests from other clients are held by the block before the end of the execution.

It is relatively simple to use a redis transaction, which has a command multi that indicates the beginning of the transaction, and then commits using exec.

There are two things to note that all commands after exec will not be executed when multi is not executed. By default, redis will temporarily store the user's next submission together in queue after receiving the multi command, and will not execute all the commands in the queue until it receives the exec command. Also, in order to improve performance, some redis clients temporarily store all commands for multi and exec locally on the redis client, and then submit them at once. This is actually based on redis's pipeline.

You can also use pipeline alone instead of using multi or exec transactions. It's just to reduce the number of redis key transmissions. This can be done if there are no data consistency problems.

When it comes to redis transactions, we have to mention the performance of redis transactions, so now combined with redis to develop distributed transactions to × × energy is also a very common solution: https://github.com/Plen-wang/redis-lock for reference.

Two persistence principles of 3.redis (RDB, AOF)

Redis supports two ways to persist data, the first: snapshotting (mirror or snapshot), also known as RDB, and the second: AOF (append-only file file append).

RDB: the mirror mode is to write all the memory data in a certain period of time directly to the hard disk.

AOF: incrementally copy the executed write commands to the hard disk.

In fact, these two are different emphasis. RDB is data persistence, AOF is command persistence, data persistence is more direct, and you can restore data directly during restore. Command persistence recovery requires a command to be executed.

The persistence operation performed by redis depends on two aspects: by default, it is performed according to the persistence configuration, and it is triggered manually by the user. There are two commands to trigger manually:

SAVE: will block all user actions until the persistence ends.

BGSAVE: the backstage child process executes. In linux, use the fork command to open a copy of the child process. This copy of the child process shares a set of memory space with the main process until the main process or child process modifies the memory, and the modified memory area is copied from the quilt process for separate use.

The problem with RDB persistence is that it is possible to lose data. AOF persistence loses commands within one second at most. So persistence combines these two to implement a balance between data integrity and performance.

The basic principle of 4.redis master-slave replication

Redis provides replication, which has many benefits. Disaster recovery, expand read and write performance.

There are two points to note: once the slave server synchronizes the data, it will empty all its own data. Redis does not support primary master replication.

The replication process is roughly as follows:

1. Connect to the master server from the slave server and send the SYNC command.

two。 The master server executes the BGSAVE and uses the cache to record all write commands executed after BGSAVE.

After the 3.BGSAVE execution is completed, the snapshot file is sent to the slave server. Discard all this instance data from the node. Load the snapshot data sent by the primary server.

4. After the snapshot is sent, the write command in the cache is sent. Perform an incremental copy from the node as usual.

5. Begin to enter the normal replication operation.

There is a problem to share, once the memory occupied by redis, even if we know key, this part of the memory can not be returned to the operating system, this is the design strategy of redis. Although the amount of memory viewed from the redis info command is useless, the operating system cannot use it.

Another problem is that if all redis servers want to replicate data, that is, to perform BGSAVE, then at least 30% of the free memory needs to be reserved. Because copy key may be needed when BGSAVE is executed, you need to leave some space in the cache if you write too many commands at this time.

5.redis extended read performance

It is actually easier to extend read performance with master-slave replication. Using the master-slave chain, the synchronization operation is assigned to the synchronization node, so that the master node can only be responsible for writing command processing.

Figure 2:

The replication node is dedicated to handle the replication function, and the downstream read node is dedicated to accept read requests. If you consider the downtime of the primary node, you can turn on the persistence function of the replication node, or turn on some nodes of the read node. The main purpose is to prevent downtime and restore master quickly. If it is a remote double activity, you can persist all the read nodes on the left and right, with a computer room on the left and a computer room on the right, which can be either double-active or restored. (the actual situation is not that simple, it's just an idea.)

The read node cannot execute the write command, which ensures future data replication.

6.redis HA Scheme (sentinel)

Redis's high availability scheme is managed based on its own set of sentinel clusters.

Figure 3:

Sentinel cluster maintains monitoring of the redis cluster, and if sentinel-1 finds that master does not respond to ping commands within a certain period of time, it subjectively concludes that it may be down. If both sentinel-2 and sentinel-3 find that master is unresponsive, then the three votes conclude that master is objectively down, and do a master / slave switch. At the same time, some notification actions are performed through the sentinel-sh script.

Of course, there are several redis-HA schemes, and we can also implement them with keepalived and VIP, separating master, backup and slave, and automatically switching VIP between master and backup.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.