Shulou(Shulou.com)06/01 Report--

Redis concept

Redis is an open source (BSD licensed), in-memory data structure storage system that can be used as database, cache, and messaging middleware. It supports many types of data structures, such as string (strings), hash (hashes), list (lists), set (sets), ordered set (sorted sets) and range query, bitmaps, hyperloglogs and geospatial (geospatial) index radius query. Redis has built-in replication (replication), LUA script (Lua scripting), LRU driven event (LRU eviction), transaction (transactions) and different levels of disk persistence (persistence), and provides high availability (high availability) through Redis Sentinel (Sentinel) and automatic partitioning (Cluster).

Redis advantage

1. High speed

Official data read and write performance 100000 / s, the reason: the database is stored in memory, redis is written in C language, and the single-threaded architecture prevents multithreaded competition.

two。 Data structure server based on key-value pair

Almost all programming languages provide dictionary-like functions, similar to this way of organizing data is called key-value-based way, unlike many key-value pairs database, the value in Redis can not only be a string, but also can be a specific data structure, which can not only facilitate the development in many application scenarios, but also improve the development efficiency. The full name of Redis is REmote Dictionary Server, which mainly provides five kinds of data structures: string, hash, list, set and ordered set. At the same time, two magical "data structures" of bitmap (Bitmaps) and HyperLogLog have evolved on the basis of string, and with the continuous development of LBS (Location Based Service).

3. Rich featur

In addition to the five data structures, Redis provides many additional features:

Provides a key expiration function, which can be used to implement caching.

The publish and subscribe function is provided, which can be used to implement the message system.

Support Lua scripting function, you can use Lua to create new Redis commands.

It provides a simple transaction function, which can guarantee the transaction characteristics to a certain extent.

The Pipeline function is provided so that the client can pass a batch of commands to the

Redis reduces the overhead of the network.

4. Simple and stable

The simplicity of Redis is mainly manifested in three aspects. First of all, there is little source code for Redis. Secondly, Redis uses a single-threaded model. Finally, Redis does not need to rely on class libraries in the operating system (for example, Memcache needs to rely on system class libraries such as libevent). Redis implements the relevant functions of event handling.

5. There are many client languages

Redis provides a simple TCP communication protocol, and many programming languages can easily access Redis.

6. Persistence

Generally speaking, it is not safe to keep data in memory, and important data may be lost in the event of a power outage or machine failure, so Redis provides two persistence methods: RDB and AOF, that is, you can use two strategies to save the data in memory to the hard disk (as shown in figure 1-1), thus ensuring the persistence of the data.

7. Master-slave replication

Redis provides replication capabilities, which implement multiple Redis copies of the same data (shown in figure 1-2), which is the basis of distributed Redis.

8. Highly available and distributed

Redis officially provides a high-availability implementation Redis Sentinel from version 2.8, which can guarantee fault detection and automatic fault transfer of Redis nodes. Redis officially provides the distributed implementation Redis Cluster from version 3.0. it is the true distributed implementation of Redis, which provides high availability, read-write and capacity scalability.

Introduction to Redis Cluster

A Redis cluster is an assembly that provides data sharing among multiple Redis nodes. Redis clusters do not support commands to handle multiple keys, because this requires moving data between different nodes, thus not achieving the performance of Redis, which can lead to unpredictable errors under high load.

Redis clusters provide a certain degree of availability through partitioning and continue to process commands when a node is down or unreachable in the real environment. The advantage of Redis cluster: automatically split the data to different nodes. Some nodes in the entire cluster can continue to process commands if they fail or are unreachable.

Data fragmentation of Redis Cluster

Redis cluster does not use consistent hash, but introduces the concept of hash slot. Redis cluster has 16384 hash slots, and each key has 16384 slots after CRC16 verification to decide which slot to place. Each node of the cluster is responsible for part of the hash slot. For example, if the current cluster has three nodes, then:

Node A contains hash slots 0 to 5500.

Node B contains hash slots 5501 to 11000.

Node C contains hash slots 11001 to 16384.

This structure is easy to add or remove nodes. For example, if I want to add a new node D, I need to get some slots from nodes A, B, C to D. If I want to remove node A, I need to move the slots in A to B and C nodes, and then remove A nodes without any slots from the cluster. Since moving hash slots from one node to another does not stop service, no matter adding, deleting or changing the number of hash slots of a node will not cause the cluster to be unavailable.

Master-Slave replication Model of Redis Cluster

In order to make the cluster still available when some nodes fail or most nodes can not communicate, the cluster uses the master-slave replication model, and each node will have a replica.

In our example, if node B fails in the absence of a replication model, the whole cluster will be unavailable because of the lack of slots in the range of 5501-11000.

However, when the cluster is created (or after a period of time), we add a slave node, A1 Magi B1, C1 for each node, then the whole cluster consists of three master nodes and three slave nodes, so that after node B fails, the cluster will elect B1 to continue to serve the new master node, and the whole cluster will not be unavailable because the slot cannot be found.

However, when both B and B 1 fail, the cluster is not available.

Redis consistency guarantee

Redis does not guarantee strong consistency of data. This means that in practice, the cluster may lose write operations under certain conditions. The first reason is that the cluster uses asynchronous replication. Write operation: the client writes a command to the master node B. Master node B returns the command status to the client. The master node copies the write operation to him and has to slave nodes B 1, B 2 and B 3. The copy of the command by the master node occurs after the command reply is returned, because if each command request needs to wait for the copy operation to complete, then the speed at which the master node processes command requests will be greatly reduced-we have to make a tradeoff between performance and consistency. Note: Redis clusters may provide synchronous write methods in the future. Another situation where commands may be lost in a Redis cluster is when a network partition appears in the cluster and a client is isolated from a small number of instances, including at least one master node.

For example, suppose the cluster consists of six nodes A, B, C, A1, B1 and C1, in which A, B, C are master nodes, A1, B1, and C1 are the subordinate nodes of A & Magi Bonomer C. there is also a client Z1 that assumes that network partitioning occurs in the cluster, then the cluster may be divided into two sides, and most of them contain nodes A, C, A1, B1 and C1. A small part of the party, including node B and client Z1.Z1, can still write to master node B. if the network partition occurs for a short time, the cluster will continue to operate normally. If the partition time is long enough for most of the party to elect B1 as the new master, then the data written by Z1 to B will be lost.

Note that during the network split, there is a limit to the maximum time that client Z1 can send write commands to master node B. this time limit is called node timeout (node timeout), which is an important configuration option for Redis clusters.

High concurrency of Redis

1.redis is based on memory, and the read and write speed of memory is very fast

2.redis is single-threaded, saving a lot of time for context switching threads.

3.redis uses multiplexing technology to handle concurrent connections. The internal implementation of non-blocking IO adopts epoll and adopts a simple event framework implemented by epoll+ itself. Read, write, close, and connect in epoll are all converted into events, and then take advantage of the multiplexing feature of epoll to never waste any time on io.

The following focuses on the reasons for the speed of single-thread design and IO multiplexing core design

Why Redis is single threaded

1. Official answer

Because Redis is a memory-based operation, CPU is not the bottleneck of Redis, and the bottleneck of Redis is most likely to be the size of machine memory or network bandwidth. Since single-threading is easy to implement and CPU will not be a bottleneck, it makes sense to adopt a single-threaded solution.

two。 Performance index

With regard to the performance of redis, official data processes hundreds of thousands of requests per second.

3. Detailed reasons

1) No performance consumption of various locks is required

The data structures of Redis are not all simple Key-Value, but also complex structures such as list,hash, which may perform very fine-grained operations, such as adding an element after a long list and adding or deleting an object in the hash. These operations may require a lot of locks, resulting in a significant increase in synchronization overhead.

In short, in the case of a single thread, there is no need to consider all kinds of locks, there is no lock release operation, and there is no performance consumption caused by possible deadlocks.

2) single-thread multi-process cluster scheme

The power of single thread is actually very powerful, and the efficiency of each core is also very high, so multi-thread can naturally have a higher performance limit than single-thread, but in today's computing environment, even the upper limit of single-machine multi-thread can not meet the needs. What needs to be further explored is the scheme of multi-server clustering, and the technology of multi-thread in these schemes is still useless. Therefore, single-threaded, multi-process cluster is a fashionable solution.

3) CPU consumption

The use of single thread avoids unnecessary context switching and competition conditions, and there is no CPU consumption caused by multi-process or multi-thread switching.

But what if CPU becomes a Redis bottleneck, or you don't want to leave the other CUP cores of the server idle?

You can consider several Redis processes. Redis is a key-value database, not a relational database, and there are no constraints between the data. As long as the client can tell which key is on which Redis process.

Advantages and disadvantages of Redis single thread

1. The code is clearer, the processing logic is simpler, there is no need to consider various locks, there is no lock release operation, and there is no CPU consumption due to the performance consumption caused by possible deadlocks, there is no switching caused by multi-processes or multi-threads.

The drawback of single process and single thread cannot give full play to the performance of multi-core CPU, but it can be improved by opening multiple Redis instances on a single machine.

2. IO Multiplexing Technology redis adopts network IO multiplexing technology to ensure the high throughput of the system in the case of multiple connections.

Multiplexing-refers to multiple socket connections, reuse-refers to the reuse of a thread. There are three main techniques for multiplexing: select,poll,epoll. Epoll is the latest and best multiplexing technology at present.

Here, "multiplex" refers to multiple network connections, and "multiplexing" refers to the reuse of the same thread. Using multi-channel IO multiplexing technology allows a single thread to process multiple connection requests efficiently (minimizing the time consumption of network Redis), and Redis operates data very fast in memory (the operation in memory will not become the performance bottleneck here). The above two points make Redis have high throughput.

Redis cluster migration

There are two methods for redis cluster migration:

1. The migration scheme uses cluster's own cluster management tool redis-trib.rb to carry out lossless migration scheme.

2. The migration scheme uses Redis cluster's own management tools to stop the old cluster nodes and archive them through rdb files.

The first way is to introduce:

Deploy the new node redis instance. (note that after deployment, there is no need to create a cluster.)

Add the new node redis to the old cluster. (add master node)

Add the new node redis to the old cluster. (add slave node)

Cluster for migration

Evenly slice the slot to the new remaining master node

Advantage: ensuring data integrity

Bug: the source redis cluster is not available after migration.

The second way is to introduce:

Old cluster nodes generate RDB files (operation premise: application disconnects Redis connection)

Get the slot information of the old cluster node and record it

Deploy a new Redis cluster migration environment

Assign slots to the new Redis cluster (corresponding to the slots of the old cluster nodes)

Copy the rdb files of the old cluster under the same slot node.

Advantages: ensure the availability of the source redis, and the application can be switched at any time.

Defect: the slot must be corresponding all the time during the migration.

Keepalived

What is Keepalived? keepalived can tell by its name, stay alive, and stay online in the network, that is, the so-called high availability or hot backup, to prevent a single point of failure (a single point of failure means that the failure of a certain point will lead to the unavailability of the entire system architecture).

Redis Master-Slave + Keepalive + VIP.

This is a classic db architecture, and you can also use master-slave switching with mysql. The basic principle is that Keepalive detects the survival of master through a script, and then completes the master-slave switch by drifting VIP (Virtual IP).

Infrastructure diagram

Basic construction and principle

1) Keepalive + VIP: deploy keepalived, redis instance survival detection script, and alarm notification script on redis master-slave.

2) when redis master fails, VIP drifts from master to slave to complete the Mmurs role and configuration changes.

3) host is set to VIP in the parameters of client connection redis, and the whole switching process is transparent to the client.

Advantages and disadvantages and applicable scenarios

Advantages: easy to implement and low cost.

Cons: the maximum throughput of the entire cluster is limited by the processing power of a single instance of redis, unless an application uses multiple sets of this Keepalive+VIP scheme.

Therefore, the scalability is poor, and it is not suitable for the current deployment scenario of deploying multiple redis instances on a single machine, while we deploy at least 8 redis instances on one physical machine.

Suitable for scenarios: applications with low concurrent requests.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.