Shulou(Shulou.com)05/31 Report--

What is Redis and its role, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. What is Redis? What is it mainly used for?

Redis, the English full name is Remote Dictionary Server (remote Dictionary Service), is an open source log database written in ANSI C language, supporting network, memory-based and persistent, Key-Value database, and provides API in multiple languages. [related recommendation: Redis video tutorial]

Unlike MySQL databases, Redis data is stored in memory. Its read and write speed is very fast and can handle more than 100000 read and write operations per second. Therefore, redis is widely used in caching. In addition, Redis is often used to do distributed locks. In addition, Redis supports transactions, persistence, LUA scripting, LRU-driven events, and multiple clustering scenarios.

two。 Talk about the basic data structure types of Redis

As most partners know, Redis has the following five basic types:

String (string)

Hash (hash)

List (list)

Set (collection)

Zset (ordered set)

It also has three special data structure types.

Geospatial

Hyperloglog

Bitmap

2.1 five basic data types of Redis

String (string)

Introduction: String is the most basic data structure type of Redis. It is binary secure and can store pictures or serialized objects. The maximum value is 512m.

Examples of simple use: set key value, get key, etc.

Application scenarios: shared session, distributed lock, counter, current limit.

There are three internal codes, int (8-byte long integer) / embstr (39-byte string less than or equal to 39 bytes) / raw (greater than 39-byte string)

The string in C language is implemented by char [], while Redis is encapsulated with SDS (simple dynamic string). The sds source code is as follows:

Struct sdshdr {unsigned int len; / / length of tag buf unsigned int free; / / number of unused elements in tag buf char buf []; / / pits where elements are stored}

The structure diagram of SDS is as follows:

Why did Redis choose the SDS structure, while the C language native char [] is not fragrant?

For example, in SDS, the length of the string can be obtained by O (1) time complexity, while the C string needs to traverse the entire string with a time complexity of O (n)

Hash (hash)

Introduction: in Redis, the hash type refers to v (value) itself is also a key-value pair (kmurv) structure

Examples of simple use: hset key field value, hget key field

Internal coding: ziplist (compressed list), hashtable (hash table)

Application scenarios: cache user information, etc.

Note: if the development uses hgetall, and there are many hash elements, it may cause Redis blocking, so you can use hscan. If you only get part of the field, it is recommended to use hmget.

The comparison of string and hash type is as follows:

List (list)

Summary: the list type is used to store multiple ordered strings, and a list can store up to 2 ^ 32-1 elements.

Simple and practical examples: lpush key value [value...], lrange key start end

Internal coding: ziplist (compressed list), linkedlist (linked list)

Application scenario: message queue, article list

One picture understands the insertion and pop-up of the list type:

For list application scenarios, please refer to the following:

Lpush+lpop=Stack (stack)

Lpush+rpop=Queue (queue)

Lpsh+ltrim=Capped Collection (finite set)

Lpush+brpop=Message Queue (message queuing)

Set (collection)

Summary: the set type is also used to save multiple string elements, but repeating elements are not allowed.

Examples of simple use: sadd key element [element...], smembers key

Internal coding: intset (set of integers), hashtable (hash table)

Note: smembers and lrange, hgetall are relatively heavy commands, if too many elements have the possibility of blocking Redis, you can use sscan to complete.

Application scenarios: user tags, random lottery generation, social needs.

Ordered set (zset)

Introduction: a collection of sorted strings, while elements cannot be repeated

Simple format example: zadd key score member [score member...], zrank key member

Underlying internal coding: ziplist (compressed list), skiplist (jump table)

Application scenarios: rankings, social needs (such as likes by users).

2.2 three special data types of Redis

Introduced by Geo:Redis3.2, geolocation is used to store geographic location information and operate on the stored information.

HyperLogLog: a data structure used for cardinality statistics algorithms, such as the UV of a statistics website.

Bitmaps: use a bit to map the state of an element. In Redis, its underlying layer is based on the string type, and the bitmaps can be turned into an array of bits.

3. Why is Redis so fast?

3.1 implementation based on memory storage

We all know that memory read and write is much faster than on disk, Redis based on memory storage implementation of the database, compared with the data stored in the disk MySQL database, save the disk Icano consumption.

3.2 efficient data structures

We know that in order to improve efficiency, Mysql index chooses the data structure of B+ tree. In fact, a reasonable data structure can make your application / program faster. First, take a look at the data structure of Redis & the internal coding diagram:

SDS simple dynamic string

String length processing: Redis acquires string length with a time complexity of O (1), while in C language, it needs to be traversed from scratch with a complexity of O (n).

Space pre-allocation: the more frequent string modification, the more frequent memory allocation, which will consume performance, while SDS modification and space expansion will allocate additional unused space and reduce performance loss.

Inert space release: when the SDS is shortened, instead of reclaiming the excess memory space, the free records the excess space, and if there are subsequent changes, directly use the space recorded in the free to reduce allocation.

Binary security: Redis can store some binary data. In C language, a string ends when it encounters'\ 0', while in SDS, it is the len attribute that marks the end of a string.

Dictionaries

Redis as a KMel V-type in-memory database, all the keys are stored in a dictionary. A dictionary is a hash table, such as HashMap, and the corresponding value can be obtained directly through key. As for the characteristics of the hash table, the corresponding value can be obtained at O (1) time complexity.

Jump table

Jump table is a unique data structure of Redis, which is to increase multi-level index to improve search efficiency on the basis of linked list.

The jump table supports node lookup with average O (logN) and worst O (N) complexity, and can also batch process nodes through sequential operations.

3.3 reasonable data coding

Redis supports multiple data types, each basic type, and possibly multiple data structures. When, what data structure and what encoding are used is the result of redis designers' summary and optimization.

String: if you store numbers, you use int type encoding; if you store non-numeric strings that are less than or equal to 39 bytes, if the embstr; is greater than 39 bytes, it is raw encoding.

List: if the number of elements in the list is less than 512, the value of each element in the list is less than 64 bytes (default), use ziplist encoding, otherwise use linkedlist encoding

Hash: if the number of hash type elements is less than 512, use ziplist encoding if all values are less than 64 bytes, otherwise use hashtable encoding.

Set: if all the elements in the collection are integers and the number of elements is less than 512, use intset encoding, otherwise use hashtable encoding.

Zset: use ziplist encoding when the number of elements in an ordered set is less than 128and the value of each element is less than 64 bytes, otherwise use skiplist (Jump Table) encoding

3.4 A reasonable threading model

Ipaw O Multiplexing

The multiplexing technology of multi-channel epoll O enables a single thread to process multiple connection requests efficiently, while epoll is used as the implementation of the multiplexing technology. Moreover, Redis's own event handling model converts the connection, read and write, and shutdown in epoll into events, and does not waste too much time on the network Ibino.

What is Icano multiplexing?

ICompo: network Ihamo

Multiplex: multiple network connections

Reuse: reuse the same thread.

IO multiplexing is actually a synchronous IO model that implements that a thread can monitor multiple file handles; once a file handle is ready, it can notify the application to read and write accordingly; and when no file handle is ready, it blocks the application and hands over the cpu.

Single thread model

Redis is a single-threaded model, which avoids the unnecessary context switching and competitive lock consumption of CPU. Also because it is single-threaded, if a command executes too long (such as the hgetall command), it will cause blocking. Redis is a database for fast execution scenarios. So be careful with commands such as smembers and lrange, hgetall, etc.

Redis 6.0introduces multithreading acceleration, which executes commands and operates on memory using a single thread.

3.5 Virtual memory mechanism

Redis directly built its own VM mechanism, unlike the general system will call system functions to deal with, will waste a certain amount of time to move and request.

What is the virtual memory mechanism of Redis?

The virtual memory mechanism is to temporarily swap infrequently accessed data (cold data) from memory to disk, thus freeing up valuable memory space for other data that needs to be accessed (hot data). The hot and cold data can be separated by the VM function, so that the hot data is still in memory and the cold data is saved to disk. In this way, the problem of slow access speed caused by insufficient memory can be avoided.

4. What is cache breakdown, cache penetration, cache avalanche? 4.1 cache penetration issues

First, let's take a look at a common cache usage: when a read request comes, check the cache first, and return directly if the cache is hit; if the cache fails, check the database, then update the database value to the cache, and then return.

Cache traversal: refers to querying a data that must not exist. Because the cache misses, it needs to be queried from the database, and if the data cannot be found, it will not be written to the cache. This will cause the non-existent data to go to the database for query every time, thus putting pressure on the database.

To put it popularly, when a read request is accessed, neither the cache nor the database has a certain value, which causes every query request for this value to penetrate into the database, which is called cache penetration.

Cache penetration is generally caused by these situations:

Unreasonable design of the business, for example, most users do not have daemons, but every request you make is cached to query whether a userid query has a daemon.

Operations / operations / development failures, such as caching and database data, have been mistakenly deleted.

Illegal request attacks by hackers, such as hackers deliberately fabricate a large number of illegal requests to read business data that does not exist.

How to avoid cache penetration? There are usually three ways.

1. If it is an illegal request, we check the parameters and filter the illegal values at the API entry.

two。 If the query database is empty, we can set a null value or a default value for the cache. However, if there are write requests coming in, you need to update the cache to ensure cache consistency, and finally set the appropriate expiration time for the cache. (more commonly used in business, simple and effective)

3. Use a Bloom filter to quickly determine whether the data exists. That is, when a query request comes, the Bloom filter is used to determine whether the value exists before continuing to look up.

Bloom filter principle: it consists of an array of bitmaps with an initial value of 0 and N hash functions. A N hash algorithm for a key to obtain N values, hash the N values in the bit array and set them to 1, and then check if the specific positions are all 1, then the Bloom filter determines the existence of the key.

4.2 cache snow Ben problem

Cache Mercedes-Benz: refers to a large number of data in the cache to the expiration time, and the huge amount of query data, requests directly access the database, causing excessive pressure on the database and even downmachine.

Cache snow Ben is generally caused by a large number of data expired at the same time, which can be solved by evenly setting the expiration time, that is, making the expiration time relatively discrete. Such as using a larger fixed value + a smaller random value, 5 hours + 0 to 1800 seconds.

The downtime of Redis failure may also cause the cache to run. This requires the construction of Redis high-availability clusters.

4.3 cache breakdown issu

Cache breakdown: when the hotspot key expires at a certain point in time, and there are a large number of concurrent requests to the Key at this time, so that a large number of requests reach the db.

Cache breakdown looks a bit like it, but the difference between it is that cache Mercedes-Benz means that the database is under too much pressure or even download. cache breakdown is only a large number of concurrent requests to the DB database level. You can think of breakdown as a subset of the cache snow Ben. Some articles think that the difference between the two is that it breaks down the key cache for a certain hot spot, while Xie Ben is a lot of key.

There are two solutions:

1. Use the mutex scheme. When the cache fails, instead of loading db data immediately, use some atomic operation commands with successful returns, such as (setnx of Redis) to operate, and then load db database data and set cache when successful. Otherwise, retry to get the cache.

two。 "never expire" means that when the expiration time is not set, but the hot spot data is about to expire, the asynchronous thread updates and sets the expiration time.

5. What is the hot Key problem and how to solve the hot key problem

What is a hot Key? In Redis, we call the frequently visited key as the hotspot key.

If a hotspot key requests to the server host, due to the large number of requests, it may lead to insufficient host resources, or even downtime, thus affecting the normal service.

And how does hot Key come into being? There are two main reasons:

The data consumed by users is much larger than the production data, such as second kill, hot news and other scenarios with more reading and less writing.

Request sharding is centralized, which exceeds the performance of a single Redi server. For example, when a fixed-name key,Hash falls into the same server, the instant traffic is extremely large, which exceeds the machine bottleneck, resulting in hot Key problems.

So how to identify hot key in daily development?

Judge which is hot Key by experience

Client statistics reporting

Service agent level report

How to solve the hot key problem?

Redis cluster expansion: add shard replicas and balance read traffic

Distribute hot key to different servers

Use a secondary cache, the JVM local cache, to reduce read requests from Redis.

6. Redis expiration policy and memory obsolescence strategy

6.1 expiration policy for Redis

When we are in set key, we can set an expiration time for it, such as expire key 60. After specifying this key60s, it expires. After 60s, how does redis handle it? Let's start with several expiration policies:

Timed expiration

Each key that sets the expiration time needs to create a timer, and the key will be cleared immediately when it expires. This strategy can clean up expired data immediately and is memory-friendly, but it takes up a lot of CPU resources to process expired data, thus affecting the response time and throughput of the cache.

Inert expiration

It is only when a key is accessed that it is determined whether the key has expired or is cleared when it expires. This strategy maximizes the savings in CPU resources, but is very memory-friendly. In extreme cases, there may be a large number of expired key that are not accessed again, so that they are not cleared and take up a lot of memory.

Periodic expiration

At regular intervals, a certain number of expires dictionaries in a certain number of databases are scanned for a certain number of key, and the expired key is cleared. This strategy is a compromise between the first two. By adjusting the time interval of timing scanning and the limited time of each scan, the optimal balance between CPU and memory resources can be achieved under different conditions.

The expires dictionary holds expiration time data for all key with an expiration time set, where key is a pointer to a key in the key space, and value is the expiration time represented by the millisecond precision UNIX timestamp of that key. The key space refers to all the keys saved in the Redis cluster.

Both lazy expiration and periodic expiration are used in Redis.

Assuming that Redis currently stores 300000 key, and all of them have an expiration time set, if you check all the key,CPU loads every other 100ms, it will be particularly high and may eventually fail.

Therefore, redis takes periodic expiration, and randomly selects a certain number of key every 100ms to check and delete.

However, in the end, there may be many expired key that have not been deleted. At this point, redis uses lazy deletion. When you get a key, redis will check that if the key is set to expire and has expired, it will be deleted.

However, if you delete regularly and miss a lot of expired key, then you don't delete it lazily. There will be a lot of expired key accumulated in memory, which will directly cause the memory to burst. Or sometimes, the volume of business becomes large, the key of redis is heavily used, the memory is directly insufficient, and the younger brother of the operation and maintenance staff also forgot to increase the memory. Did redis just hang up like this? It won't! Redis protects itself with 8 memory elimination strategies.

6.2 Redis memory obsolescence strategy

Volatile-lru: when there is not enough memory to hold new writes, use the LRU (least recently used) algorithm to eliminate it from the key with the expiration time set

Allkeys-lru: when there is not enough memory to hold new writes, use the LRU (least recently used) algorithm to eliminate it from all key.

The volatile-lfu:4.0 version is new, and when there is not enough memory to hold the newly written data, the LFU algorithm is used to delete the key in the expired key.

The allkeys-lfu:4.0 version has been added to eliminate the LFU algorithm from all key when there is not enough memory to accommodate new writes.

Volatile-random: when there is not enough memory to hold the newly written data, the data is randomly eliminated from the key with the expiration time set.

Allkeys-random: randomly eliminates data from all key when there is not enough memory to hold new writes.

Volatile-ttl: when there is not enough memory to hold new write data, in the key with the expiration time set, the expiration time is eliminated according to the expiration time. The earlier the expiration is, the priority is eliminated.

Noeviction: the default policy is that new writes report errors when there is not enough memory to hold new writes.

7. Talk about the common application scenarios of Redis

Caching

Ranking

Counter application

Share Session

Distributed lock

Social Network

Message queue

Bit operation

7.1 cach

When we mention redis, we naturally think of caching. Large and medium-sized websites at home and abroad can not do without caching. Reasonable use of caching, such as caching hot data, can not only improve the access speed of the website, but also reduce the pressure on the database DB. Moreover, compared with memcached, Redis also provides rich data structures and persistence mechanisms such as RDB and AOF.

7.2 ranking

Today's Internet applications, there are a variety of rankings, such as monthly sales rankings of e-commerce sites, social APP gift rankings, Mini Program voting rankings and so on. The zset data type provided by Redis can achieve these complex rankings.

For example, the ranking of users uploading videos every day and getting likes can be designed as follows:

1. User Jay uploads a video and gets 6 likes, which can be as follows:

Zadd user:ranking:2021-03-03 Jay 3

After a while, get another like, you can do this:

Zincrby user:ranking:2021-03-03 Jay 1

If a user John cheats, you need to delete the user:

Zrem user:ranking:2021-03-03 John

Show the three users who received the most likes

Application of zrevrangebyrank user:ranking:2021-03-03 027.3 counter

Major websites and APP applications often need counter functions, such as the number of short videos played and the number of visits to e-commerce websites. These playbacks and views generally require real-time, each playback and browsing have to do plus 1 operation, if the concurrency is very large for the performance of traditional relational data is a challenge. Redis naturally supports the counting function and the counting performance is also very good, which can be said to be an important choice for the counter system.

7.4 shared Session

If a distributed Web service stores the user's Session information on its own server, the user may need to log in again after a refresh, which is obviously a problem. In fact, you can use Redis to centrally manage a user's Session, and each user updates or queries login information directly from the Redis.

7.5 distributed locks

Distributed deployment is used in almost every Internet company. under distributed services, we will encounter the technical problems of concurrent access to the same resource, such as seconds kill, issuing orders to reduce inventory and other scenarios.

It is definitely not possible to use synchronize or reentrantlock local locks.

If the amount of concurrency is small, it is no problem to use pessimistic locks and optimistic locks of the database.

However, in the case of high concurrency, the use of database locks to control concurrent access to resources will affect the performance of the database.

In fact, you can use Redis's setnx to implement distributed locks.

7.6 Social networking

Likes / likes, fans, mutual friends / preferences, push, drop-down refresh and so on are essential functions of social networking sites. Because social networking sites usually have a large number of visitors, and traditional relational data is not suitable for storing this type of data, the data structure provided by Redis can relatively easily achieve these functions.

7.7 message queuing

Message queuing is a necessary middleware for large websites, such as ActiveMQ, RabbitMQ, Kafka and other popular message queuing middleware, which is mainly used for business decoupling, traffic peaking and low real-time asynchronous processing. Redis provides publish / subscribe and blocking queue functions to implement a simple message queuing system. In addition, this cannot be compared with professional messaging middleware.

7.8 bit operation

It is used in scenarios with hundreds of millions of data, such as check-in of hundreds of millions of user systems, statistics of re-login times, whether a user is online, and so on. Tencent 1 billion users, in a few milliseconds to find out whether a user is online, what can be done? Don't tell me to create a key for each user and write it down one by one (you can calculate that the memory you need is scary, and there are a lot of similar needs. Here you want to use the in place operation-- use the setbit, getbit, bitcount commands. The principle is: redis to build a long enough array, each array element can only be 0 and 1 two values, and then the array subscript index is used to represent the user id (must be a number ha), then it is obvious that this hundreds of millions of long array can be subscript and element values (0 and 1) to build a memory system.

8. What are the persistence mechanisms of Redis? Talk about the advantages and disadvantages.

Redis is a memory-based non-relational Kmuri V database, and since it is memory-based, if the Redis server dies, the data will be lost. To avoid data loss, Redis provides persistence, that is, saving the data to disk.

Redis provides two persistence mechanisms: RDB and AOF. The persistence file loading process is as follows:

8.1 RDB

RDB is to save memory data to disk in the form of snapshots.

What is a snapshot? It can be understood this way, take a picture of the current moment's data, and then save it.

RDB persistence means that the snapshot of the dataset in memory is written to disk by performing a specified number of writes within a specified time interval. It is the default persistence method of Redis. After the operation is completed, a dump.rdb file is generated in the specified directory. When Redis restarts, the data is recovered by loading the dump.rdb file. The main triggering mechanisms of RDB are as follows:

Advantages of RDB

Suitable for large-scale data recovery scenarios, such as backup, full replication, etc.

Shortcomings of RDB

There is no way to achieve real-time persistence / second-level persistence.

There is a problem of RDB format compatibility between the new and old versions.

AOF

AOF (append only file) persistence records each write operation in the form of a log, appends it to the file, and reexecutes the commands in the AOF file when restarting to recover the data. It mainly solves the real-time problem of data persistence. It is not enabled by default.

The workflow of AOF is as follows:

Advantages of AOF

Higher consistency and integrity of data

Shortcomings of AOF

The more AOF records, the larger the file, and the slower the data recovery.

9. How to achieve high availability of Redis?

When we use Redis in our project, we certainly will not deploy the Redis service at a single point. Because, once a single point of deployment goes down, it is not available. In order to achieve high availability, it is common practice to replicate multiple copies of the database for deployment on different servers, and one of them can continue to provide services after it has hung up. There are three deployment modes for Redis to achieve high availability: master-slave mode, Sentinel mode, and cluster mode.

9.1 Master-Slave Mode

In the master-slave mode, Redis deploys multiple machines, has master nodes, is responsible for read and write operations, and has slave nodes, which are only responsible for read operations. The data of the slave node comes from the master node, and the implementation principle is the master-slave replication mechanism.

Master-slave replication includes full replication and incremental replication. Generally speaking, when slave starts the connection master for the first time, or thinks it is the first time to connect, it uses full replication. The full replication process is as follows:

1.slave sends a sync command to master.

After receiving the SYNC command, 2.master executes the bgsave command to generate the full RDB file.

3.master uses buffers to record all write commands during RDB snapshot generation.

After 4.master executes the bgsave, it sends RDB snapshot files to all slave.

After 5.slave receives the RDB snapshot file, it loads and parses the received snapshot.

6.master uses buffers to record all write commands generated during RDB synchronization.

After the 7.master snapshot is sent, start sending write commands in the buffer to slave

8.salve accepts command requests and executes write commands from the master buffer

Since the redis2.8 version, psync has been used instead of sync because the sync command consumes system resources and psync is more efficient.

After slave is fully synchronized with master, if the data on master is updated again, incremental replication will be triggered.

When data increases or decreases on the master node, the replicationFeedSalves () function is triggered, and each subsequent command called on the Master node uses replicationFeedSlaves () to synchronize to the Slave node. Before executing this function, the master node determines whether the command executed by the user has a data update, and if there is a data update, and the slave node is not empty, the function will be executed. The purpose of this function is to send the commands executed by the user to all slave nodes and let the slave node execute. The process is as follows:

9.2 Sentinel mode

In the master-slave mode, once the master node is unable to provide services due to failure, it needs to manually promote the slave node to the master node, and inform the application party to update the address of the master node. Obviously, this fault handling is not acceptable to most business scenarios. Redis has officially provided the Redis Sentinel (Sentinel) architecture since 2.8to solve this problem.

Sentinel mode, a Sentinel system consisting of one or more Sentinel instances, can monitor all Redis master nodes and slaves, and automatically upgrade a slave node under the offline master server to a new master node when the monitored master node enters the offline state. However, when a sentinel process monitors Redis nodes, there may be problems (a single point of problem). Therefore, multiple sentinels can be used to monitor Redis nodes, and monitoring will be carried out among sentinels.

To put it simply, the Sentinel model serves three purposes:

Send a command and wait for the Redis server (including master server and slave server) to return to monitor its running status

When the sentry detects that the master node is down, it will automatically switch the slave node to the master node, and then notify the other slave nodes through the publish and subscribe mode, modify the configuration file and let them switch hosts.

Sentinels also monitor each other to achieve high availability.

What is the process of failover?

Assuming that the primary server is down, Sentinel 1 first detects this result, the system will not immediately carry out the failover process, only Sentinel 1 subjectively thinks that the primary server is unavailable, this phenomenon becomes subjective offline. When the subsequent sentinel also detects that the primary server is unavailable and the number reaches a certain value, then a vote will be conducted between the sentinels, and the result of the vote will be initiated by a sentry for failover operation. After the switch is successful, each sentry will switch hosts from the server monitored by himself through the publish and subscribe mode. This process is called objective offline. This way, everything is transparent to the client.

The mode of operation of the Sentinel is as follows:

Each Sentinel sends a PING command to its known Master,Slave and other Sentinel instances at a frequency of once per second.

If an instance (instance) takes longer than the value specified by the down-after-milliseconds option from the last valid reply to the PING command, the instance will be marked as subjectively offline by Sentinel.

If a Master is marked as subjectively offline, all Sentinel that are monitoring the Master should confirm that the Master has indeed entered the subjective offline state at a frequency of once per second.

When there is a sufficient number of Sentinel (greater than or equal to the value specified in the profile) to confirm that the Master has indeed entered the subjective offline state within the specified time range, the Master will be marked as objective offline.

In general, each Sentinel sends INFO commands to all Master,Slave it knows at a frequency of every 10 seconds.

When Master is marked as objective offline by Sentinel, the frequency of Sentinel sending INFO commands to all Slave of offline Master will be changed from once in 10 seconds to once per second.

If there is not enough Sentinel to agree that Master has been offline, the objective offline status of Master will be removed; if Master returns a valid reply to Sentinel's PING command, the subjective offline status of Master will be removed.

9.3 Cluster Cluster Model

Sentinel mode is based on master-slave mode to achieve read-write separation, it can also be automatically switched, the system availability is higher. But the data stored in each node is the same, which wastes memory and is difficult to expand online. Therefore, the Cluster cluster arises at the historic moment, which is joined in Redis3.0 and realizes the distributed storage of Redis. Slicing the data, that is, storing different content on each Redis node to solve the problem of online expansion. It also provides replication and failover capabilities.

Communication of Cluster Cluster Node

A Redis cluster consists of multiple nodes. How do the nodes communicate with each other? Through the Gossip protocol!

Redis Cluster cluster communicates through Gossip protocol, and nodes exchange information constantly before, including node failure, new node joining, master-slave node change information, slot information and so on. There are four kinds of commonly used Gossip messages: ping, pong, meet and fail.

Meet message: notify the new node to join. The message sender notifies the receiver to join the current cluster. After the meet message communication is completed normally, the receiving node will join the cluster and exchange ping and pong messages periodically.

Ping message: the most frequently exchanged message in the cluster. Each node in the cluster sends ping messages to multiple other nodes every second to detect whether the nodes are online and exchange status information with each other.

Pong message: when receiving ping and meet messages, reply to the sender as a response message to confirm that the message communicates normally. The pong message encapsulates its own state data. Nodes can also broadcast their own pong messages to the cluster to notify the entire cluster to update its status.

Fail message: when a node decides that another node in the cluster is offline, it broadcasts a fail message to the cluster. Other nodes receive the fail message and update the corresponding node to offline status.

In particular, each node communicates with other nodes through the Cluster bus (cluster bus). When communicating, use a special port number, that is, the external service port number plus 10000. For example, if the port number of a node is 6379, the port number that it communicates with other nodes is 16379. The communication between nodes uses a special binary protocol.

Hash Slot slot algorithm

Since it is distributed storage, is the distributed algorithm used by Cluster clusters consistent Hash? Not really, but the Hash Slot slot algorithm.

The slot algorithm divides the whole database into 16384 slot (slots). Each key-value pair entering the Redis is hashed according to key and assigned to one of the 16384 slots. The use of hash mapping is also relatively simple, using the CRC16 algorithm to calculate a 16-bit value, and then take the module for 16384. Each key in the database belongs to one of the 16384 slots, and each node in the cluster can handle the 16384 slots.

Each node in the cluster is responsible for a part of the hash slots. For example, the current cluster has A, B, C nodes, and the number of hash slots on each node is 16384big 3, then there are:

Node An is responsible for hash slot No. 5460.

Node B is responsible for hash slot No. 5461 to 10922.

Node C is responsible for hash slot 109230016383.

Redis Cluster cluster

In a Redis Cluster cluster, you need to ensure that the node corresponding to 16384 slots is working properly. If a node fails, the slot it is responsible for will also fail, and the whole cluster will not work.

Therefore, in order to ensure high availability, the Cluster cluster introduces master-slave replication, with one master node corresponding to one or more slave nodes. When other master nodes ping a master node A, if more than half of the master nodes communicate with A timed out, then the master node An is considered to be down. If the master node goes down, the slave node is enabled.

On each node of the Redis, there are two gadgets, one is the slot (slot), and its value range is 016383. The other is cluster, which can be understood as a plug-in for cluster management. When the key we access arrives, Redis gets a value of 16 bit according to the CRC16 algorithm, and then modulates the result to 16384. Each key will correspond to a hash slot numbered between 016383. Through this value, you can find the node corresponding to the corresponding slot, and then automatically jump to the corresponding node for access.

Although the data is stored separately on different nodes, for the client, the entire cluster Cluster is considered as a whole. The client connects to any node, which looks the same as the Redis of the operation order instance. When the key of the client operation is not assigned to the correct node node, the Redis returns the redirection instruction and finally points to the correct node, which is a bit like the 302 redirection jump of the browser page.

Fail-over

The Redis cluster achieves high availability. When the nodes in the cluster fail, the cluster provides services normally through failover.

The redis cluster realizes fault discovery through ping/pong messages. This environment includes subjective referral and objective referral.

Subjective offline: a node thinks that another node is unavailable, that is, offline state, this state is not the final fault determination, can only represent the opinion of one node, there may be misjudgment.

Objective offline: the indicator marks the real downline of a node, and multiple nodes in the cluster think that the node is not available, thus reaching a consensus. If the master node of the holding slot fails, you need to fail over for that node.

If node A marks node B as subjective offline, after a period of time, node A sends the status of node B to other nodes through messages. When node C receives the message and parses the message body, if the pfail status of node B is found, the objective offline process will be triggered.

When the primary node is offline, the Redis Cluster cluster votes for the master node of the statistical holding slot to see if the number of votes reaches half. When the offline report statistics is greater than half, it is marked as an objective offline status.

The process is as follows:

Fault recovery: after the fault is discovered, if the offline node is the master node, it needs to be replaced by one of its slave nodes to ensure the high availability of the cluster. The process is as follows:

Qualification check: check whether the slave node is qualified to replace the failed master node.

Time to prepare for the election: update the time to trigger the failure election after the qualification check is passed.

Initiate an election: when it is time for a faulty election, an election will be held.

Election voting: only the master node that holds the slot has votes, and enough votes (more than half) are collected from the node to trigger the replacement master node operation.

10. Have you ever used Redis distributed locks? What are the points for attention?

Distributed lock is a kind of lock that controls different processes in distributed system to access shared resources together. Distributed locks are required in business scenarios such as issuing orders and grabbing red packets. Redis is often used as a distributed lock in our projects.

Choose several implementation methods of Redis distributed lock, let's discuss it and see if there is any problem.

Command setnx + expire to write separately

Setnx + value value is the expiration time

Extension command for set (set ex px nx)

Set ex px nx + check the unique random value, and then delete

10.1Command setnx + expire to write if separately (jedis.setnx (key,lock_value) = = 1) {/ / lock expire (key,100); / / set expiration time try {do something / / business request} catch () {} finally {jedis.del (key); / / release lock}}

If the process crash drops or is about to restart maintenance after executing the setnx lock and is about to execute the expire to set the expiration time, the lock will be immortal, and other threads will never acquire the lock, so distributed locking cannot be implemented in this way.

10.2The setnx + value value is the expiration time long expires = System.currentTimeMillis () + expireTime; / / system time + set expiration time String expiresStr = String.valueOf (expires); / / if the current lock does not exist, if (jedis.setnx (key, expiresStr) = = 1) {return true;} / / if the lock already exists, get the expiration time of the lock String currentValueStr = jedis.get (key) / / if the expiration time obtained is less than the current time of the system, if (currentValueStr! = null & & Long.parseLong (currentValueStr) has expired

< System.currentTimeMillis()) { // 锁已过期，获取上一个锁的过期时间，并设置现在锁的过期时间（不了解redis的getSet命令的小伙伴，可以去官网看下哈） String oldValueStr = jedis.getSet(key_resource_id, expiresStr); if (oldValueStr != null && oldValueStr.equals(currentValueStr)) { // 考虑多线程并发的情况，只有一个线程的设置值和当前值相同，它才可以加锁 return true; }} //其他情况，均返回加锁失败return false;} 笔者看过有开发小伙伴是这么实现分布式锁的，但是这种方案也有这些缺点： 过期时间是客户端自己生成的，分布式环境下，每个客户端的时间必须同步。 没有保存持有者的唯一标识，可能被别的客户端释放/解锁。 锁过期的时候，并发多个客户端同时请求过来，都执行了jedis.getSet()，最终只能有一个客户端加锁成功，但是该客户端锁的过期时间，可能被别的客户端覆盖。 10.3： set的扩展命令（set ex px nx）（注意可能存在的问题）if（jedis.set(key, lock_value, "NX", "EX", 100s) == 1）{ //加锁 try { do something //业务处理 }catch(){　　}　　finally { jedis.del(key); //释放锁 }} 这个方案可能存在这样的问题： 锁过期释放了，业务还没执行完。 锁被别的线程误删。 10.4 set ex px nx + 校验唯一随机值,再删除if（jedis.set(key, uni_request_id, "NX", "EX", 100s) == 1）{ //加锁 try { do something //业务处理 }catch(){　　}　　finally { //判断是不是当前线程加的锁,是才释放 if (uni_request_id.equals(jedis.get(key))) { jedis.del(key); //释放锁 } }} 在这里，判断当前线程加的锁和释放锁是不是一个原子操作。如果调用jedis.del()释放锁的时候，可能这把锁已经不属于当前客户端，会解除他人加的锁。 一般也是用lua脚本代替。lua脚本如下： if redis.call('get',KEYS[1]) == ARGV[1] then return redis.call('del',KEYS[1]) else return 0end; 这种方式比较不错了，一般情况下，已经可以使用这种实现方式。但是存在锁过期释放了，业务还没执行完的问题（实际上，估算个业务处理的时间，一般没啥问题了）。 11. 使用过Redisson嘛？说说它的原理 分布式锁可能存在锁过期释放，业务没执行完的问题。有些小伙伴认为，稍微把锁过期时间设置长一些就可以啦。其实我们设想一下，是否可以给获得锁的线程，开启一个定时守护线程，每隔一段时间检查锁是否还存在，存在则对锁的过期时间延长，防止锁过期提前释放。 当前开源框架Redisson就解决了这个分布式锁问题。我们一起来看下Redisson底层原理是怎样的吧： 只要线程一加锁成功，就会启动一个watch dog看门狗，它是一个后台线程，会每隔10秒检查一下，如果线程1还持有锁，那么就会不断的延长锁key的生存时间。因此，Redisson就是使用Redisson解决了锁过期释放，业务没执行完问题。 12. 什么是Redlock算法 Redis一般都是集群部署的，假设数据在主从同步过程，主节点挂了，Redis分布式锁可能会有哪些问题呢？一起来看些这个流程图： 如果线程一在Redis的master节点上拿到了锁，但是加锁的key还没同步到slave节点。恰好这时，master节点发生故障，一个slave节点就会升级为master节点。线程二就可以获取同个key的锁啦，但线程一也已经拿到锁了，锁的安全性就没了。 为了解决这个问题，Redis作者 antirez提出一种高级的分布式锁算法：Redlock。Redlock核心思想是这样的： 搞多个Redis master部署，以保证它们不会同时宕掉。并且这些master节点是完全相互独立的，相互之间不存在数据同步。同时，需要确保在这多个master实例上，是与在Redis单实例，使用相同方法来获取和释放锁。 我们假设当前有5个Redis master节点，在5台服务器上面运行这些Redis实例。 RedLock的实现步骤:如下 1.获取当前时间，以毫秒为单位。 2.按顺序向5个master节点请求加锁。客户端设置网络连接和响应超时时间，并且超时时间要小于锁的失效时间。（假设锁自动失效时间为10秒，则超时时间一般在5-50毫秒之间,我们就假设超时时间是50ms吧）。如果超时，跳过该master节点，尽快去尝试下一个master节点。 3.客户端使用当前时间减去开始获取锁时间（即步骤1记录的时间），得到获取锁使用的时间。当且仅当超过一半（N/2+1，这里是5/2+1=3个节点）的Redis master节点都获得锁，并且使用的时间小于锁失效时间时，锁才算获取成功。（如上图，10s>

30ms+40ms+50ms+4m0s+50ms)

If the lock is taken, the true effective time of the key changes, and the time it takes to acquire the lock needs to be subtracted.

If the acquisition of the lock fails (the lock has not been obtained in at least one master instance, or the lock time has exceeded the valid time), the client needs to unlock on all master nodes (even if some master nodes are not locked at all, it needs to be unlocked to prevent some from getting caught).

The next steps to simplify are:

Request locking to 5 master nodes sequentially

Whether or not to skip the master node is determined by the set timeout period.

If the locking is successful for three nodes greater than or equal to, and the time used is less than the validity period of the lock, the locking can be considered successful.

If the acquisition of the lock fails, unlock it!

13. Redis's jump table

Jump table is one of the underlying implementations of ordered set zset.

The jump table supports node lookup with average O (logN) and worst O (N) complexity, and can also batch process nodes through sequential operations.

The implementation of the jump table consists of two structures: zskiplist and zskiplistNode, in which zskiplist is used to store jump table information (such as header node, footer node, length), and zskiplistNode is used to represent jump table nodes.

The jump table is to increase the multi-level index to improve the search efficiency on the basis of the linked list.

14. How to ensure double write consistency between MySQL and Redis

Cache delay double deletion

Delete cache retry mechanism

Read biglog async delete cache

14.1 delay double deletion?

What is delayed double deletion? The flow chart is as follows:

Delete the cache first

Update the database again

Sleep for a while (for example, 1 second) and delete the cache again.

This dormant for a while, usually how long? It's all one second?

This sleep time = the time it takes to read business logic data + hundreds of milliseconds. To ensure that the read request ends, the write request can delete the cache dirty data that may be caused by the read request.

This scheme is OK, only dormant for a while (for example, that one second), there may be dirty data, the general business will also accept. But what if the second deletion of the cache fails? The data in the cache and the database may still be inconsistent, right? How about setting a natural expire expiration time for Key and letting it expire automatically? So the business has to accept data inconsistencies within the expiration time? Or are there other better options?

14.2 delete cache retry mechanism

Because the delayed double deletion may cause data inconsistency due to the deletion cache failure in the second step. You can use this scheme to optimize: delete a few more times if you fail to delete, to ensure the success of deleting the cache. ~ so you can introduce the delete cache retry mechanism.

Write request to update database

Cache failed to delete for some reason

Put the deleted key into the message queue

Consume the message of the message queue and get the key to be deleted

Retry the delete cache operation

14.3 read biglog async delete cache

It's okay to try to delete the cache mechanism again, but it will cause a lot of business code intrusion. In fact, it can also be optimized to phase out key asynchronously through the binlog of the database.

Take mysql as an example.

You can use Ali's canal to send binlog log collection to the MQ queue

Then confirm that the update message is processed through the ACK mechanism, delete the cache and ensure the consistency of the data cache.

15. Why change to multithreading after Redis 6. 0?

Before Redis6.0, Redis handled client requests, including reading socket, parsing, executing, writing socket, etc., by a sequential main thread, which is called "single thread".

Why didn't you use multithreading before Redis6.0? When using Redis, it is almost impossible for CPU to become a bottleneck, and Redis is mainly limited by memory and network. For example, on an ordinary Linux system, Redis can process 1 million requests per second by using pipelining, so if the application mainly uses O (N) or O (log (N)) commands, it will hardly consume too much CPU.

Redis uses multithreading not to completely abandon single threading, redis still uses single threading model to handle client requests, only uses multithreading to handle data reading and writing and protocol parsing, and executes commands using single threading.

The purpose of this is because the performance bottleneck of redis lies in the network IO rather than CPU, and the use of multithreading can improve the efficiency of IO reading and writing, thus improving the performance of redis as a whole.

16. Talk about Redis transaction mechanism

Redis implements the transaction mechanism through a set of commands such as MULTI, EXEC, WATCH, etc. Transactions support the execution of multiple commands at a time, and all commands in a transaction are serialized. During transaction execution, commands in the queue are serialized sequentially, and command requests submitted by other clients are not inserted into the transaction execution command sequence.

In short, a Redis transaction is a sequential, one-time, exclusive execution of a series of commands in a queue.

The process for Redis to execute a transaction is as follows:

Start transaction (MULTI)

Order to join the team

Execute transaction (EXEC), undo transaction (DISCARD)

The command describes that EXEC executes commands within all transaction blocks, DISCARD cancels transactions, abandons all commands within transaction blocks, MULTI marks the beginning of a transaction block, UNWATCH cancels the WATCH command to monitor all key. WATCH monitors the key, and if the key is changed by other commands before the transaction is executed, the transaction will be interrupted. 17. What about the Hash conflict of Redis

Redis, as an in-memory database of Kmuri V, uses a global hash to hold all the key-value pairs. This hash table consists of multiple hash buckets. The entry element in the hash bucket holds the key and value pointers, where * key points to the actual key and * value points to the actual value.

Hash table lookup rate is very fast, somewhat similar to HashMap in Java, it allows us to quickly find key-value pairs in O (1) time complexity. First, calculate the hash value through key, find the corresponding hash bucket location, then locate to entry, and find the corresponding data in entry.

What is a hash conflict?

Hash conflict: through different key, the same hash value is calculated, resulting in falling in the same hash bucket.

In order to solve the hash conflict, Redis uses chain hash. Chained hash means that in the same hash bucket, multiple elements are stored in a linked list, and they are connected by pointers in turn.

Some readers may also have doubts: the elements on the hash collision chain can only be looked up and manipulated one by one through the pointer. When more data is inserted into the hash table, the more conflicts and the longer the conflict linked list, the less efficient the query.

To remain efficient, Redis does rehash operations on the hash table, that is, increasing hash buckets and reducing collisions. To make rehash more efficient, Redis also uses two global hash tables by default, one for current use, called the primary hash table, and one for expansion, called the standby hash table.

18. Can Redis process write requests at the same time during RDB generation?

Yes, Redis provides two instructions to generate RDB, which are save and bgsave.

If it is a save instruction, it will block because it is executed by the main thread.

If it is a bgsave instruction, fork a child process to write to the RDB file, snapshot persistence is completely handed over to the child process, and the parent process can continue to process the client request.

19. What protocol is used at the bottom of Redis?

RESP, the English full name is Redis Serialization Protocol, it is a set of serialization protocol specially designed for redis. This protocol actually appeared in version 1.2 of redis, but it was not until redis2.0 that it finally became the standard of the redis communication protocol.

RESP mainly has the advantages of simple implementation, fast parsing speed, good readability and so on.

20. Bloom filter

To deal with the cache penetration problem, we can use the Bloom filter. What is a Bloom filter?

Bloom filter is a small data structure, which consists of a long binary vector and a set of Hash mapping functions. It is used to retrieve whether an element is in a set. The space efficiency and query time are much better than the general algorithm, but the disadvantage is that it has a certain error recognition rate and deletion difficulties.

What is the principle of Bloom filter? Let's say we have a set A, A Using k hash functions, each element in An is mapped to different positions in an array B of length a bit, where the binary number is set to 1. If the element to be checked, after the mapping of the k hash functions, it is found that all the binary numbers in k positions are 1, this element is likely to belong to set A, on the contrary, it must not belong to set A.

Let's look at a simple example. Let's say that set A has three elements, which are {d1magentin d2magentd3}. There is a hash function, Hash2. Now map each element of A to a 16-bit array B.

Let's map D1 now. Assuming Hash2 (D1) = 2, we change the grid in array B with subscript 2 to 1, as follows:

Let's map D2 now. Suppose Hash2 (D2) = 5. Let's change the grid in array B with subscript 5 to 1, as follows:

Then we map d3, assuming that Hash2 (d3) is also equal to 2, which is also a grid index 1 with subscript 2:

Therefore, we want to confirm whether an element dn is in set A, we just need to calculate the index subscript obtained by Hash2 (dn), as long as it is 0, it means that the element is not in set A, what if the index subscript is 1? Then the element may be one of the elements in A. Because you see, the subscript values obtained by D1 and D3 may both be 1, and they may also be mapped by other numbers, and the Bloom filter has this disadvantage: there will be false positives caused by hash collisions, and there will be errors in judgment.

How to reduce this error?

Do more hash function mapping to reduce the probability of hash collision

At the same time, increasing the bit length of B array can not only increase the range of data generated by hash function, but also reduce the probability of hash collision.

Let's add a Hash3 hash mapping function, assuming that Hash3 (D1) = 6 and Hash3 (d3) = 8, the two will not conflict, as follows:

Even if there is an error, we can find that the Bloom filter does not store complete data, it just uses a series of hash mapping functions to calculate the location, and then populates the binary vector. If the number is very large, the Bloom filter achieves great savings in storage space with very few error rates, which is quite cost-effective.

At present, the Bloom filter already has the corresponding implementation of open source class libraries, such as Google's Guava class library, Twitter's Algebird class library, readily available, or based on Redis's own Bitmaps design is also possible.

After reading the above, have you mastered what Redis is and how it works? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.