Shulou(Shulou.com)06/02 Report--

Scenario: Redis interview

(the picture comes from the Internet)

Interviewer: I see on your resume that you are proficient in using Redis, so what is the use of Redis?

Xiao Ming: (secretly happy in my heart, isn't Redis just a cache? Redis is mainly used as a cache to store non-persistent data efficiently through memory

Interviewer: can Redis be used as a persistence store?

Xiao Ming: mm-hmm. It should be all right.

Interviewer: so how does Redis persist?

Xiao Ming: mm-hmm. I'm not sure.

Interviewer: what is the memory elimination mechanism of Redis?

Xiao Ming: mm-hmm. I don't know.

Interviewer: what else can we do with Redis? Which instruction of Redis is used respectively?

Xiao Ming: all I know is that Redis can also do distributed locks and message queues.

Interviewer: all right, let's move on to the next topic.

Thinking: it is obvious that Xiao Ming's performance and answer on Redis in the interview process must have failed. Redis is something we use every day in our work. Why do we lose items when it comes to the interview?

As developers, we are used to using things that have been encapsulated by the gods to ensure that we can focus more on business development, but we don't know what the underlying implementation of these common tools is, so although it is usually easy to use, but the interview still can not make the interviewer's eyes bright.

This paper summarizes some knowledge points of Redis, there are principles and applications, I hope it can help you.

What is Redis

REmote DIctionary Server (Redis) is a key-value storage system written by Salvatore Sanfilippo.

Redis is an open source log database written in ANSI, C language, complying with BSD protocol, supporting network, memory-based and persistent, Key-Value database, and providing API in multiple languages.

Here I quote the description of Redis in the Redis tutorial, which is official but standard.

Can be based on memory can also be persisted log, Key-Value database.

I think this description is very appropriate and comprehensive.

1.1 the industry status of Redis

Redis is the most widely used storage middleware in the field of Internet technology. It is widely praised because of its high performance, perfect documents, various application capabilities and rich and perfect client support. Especially, it has become the most popular middleware in the field because of its performance and reading speed. Almost every software company uses Redis, including many large Internet companies, such as JD.com, Ali, Tencent, github and so on. As a result, Redis has become an essential skill for back-end developers.

1.2 knowledge graph

In my opinion, to learn every skill, you need to have a clear context and structure, otherwise you don't know what you know and how much you haven't learned. Just like a book, if there is no catalogue chapter, it will lose its soul.

So I try to summarize the knowledge graph of Redis, also known as brain map, as shown in the following picture, the knowledge points may not be very complete, and will be constantly updated and supplemented later.

The knowledge points of this series of articles will also be basically the same as this brain map. This article first introduces the basic knowledge of Redis, and the subsequent articles will introduce in detail the data structure, application, persistence and other aspects of Redis.

Second, Redis has the advantages of 2.1 high speed.

As a caching tool, the most well-known feature of Redis is its speed. How fast is it? Redis stand-alone qps (concurrency per second) can reach 110000 times per second, and the write speed is 81000 times per second.

So why is Redis so fast?

The vast majority of requests are purely memory operations and are very fast; data structures in Redis are specially designed for data storage using data structures in which many lookup operations are particularly fast. Such as HashMap, the time complexity of search and insertion is O (1); single thread avoids unnecessary context switching and competition conditions, and there is no CPU consumption caused by multi-process or multi-thread switching, there is no need to consider all kinds of locks, there is no lock, release lock operation, there is no performance consumption caused by possible deadlock; non-blocking Icord O multiplexing mechanism is used. 2.2 Rich data types

Redis has five common data types: String, List, Hash, set, and zset, each of which has its own use.

2.3 atomicity, supporting transactions

Redis supports transactions, and all its operations are atomic, while Redis also supports the atomicity of several merged operations.

2.4 Rich features

Redis has rich features, such as being used as distributed locks, persisting data, being used as message queues, rankings, counters, and supporting publish/subscribe, notifications, key expiration, and so on. When we want to use middleware to solve practical problems, Redis can always play its part.

III. Comparison between Redis and Memcache

Memcache and Redis are both excellent and high-performance in-memory databases. When we talk about Redis, we usually compare Memcache with Redis. Why make a comparison? Of course, it is to set off how good Redis is. If there is no comparison, there will be no harm.) the aspects of comparison include:

3.1 Storage mode

Memcache stores all the data in memory and will hang up after a power outage, so the data cannot be persisted, and the data cannot exceed the memory size.

Redis has a part of the data stored on the hard disk, which can make the data persistent. 3.2 data support types

Memcache supports relatively simple data types and only supports data structures of type String.

Redis has a wealth of data types, including: String, List, Hash, Set, Zset. 3.3 underlying model used

The underlying implementation between them and the application protocol for communicating with the client are different.

Redis directly built its own VM mechanism, because the general system calls system functions, will waste a certain amount of time to move and request. 3.4 Storage value size Redis can store up to 1GB, while memcache only has 1MB.

See here, will you think that Redis is very good, all advantages, perfect? In fact, Redis still has many shortcomings, how should we overcome these shortcomings?

IV. The problems existing in Redis and its solution 4.4.The problem of double write consistency of cache database

Problem: the problem of consistency is a common problem in distributed systems. Consistency is generally divided into two types: strong consistency and final consistency. When we want to meet strong consistency, Redis can not be flawless, because database and cache double write, there are bound to be inconsistencies, Redis can only guarantee the final consistency.

Solution: how can we ensure ultimate consistency?

The first way is to set a certain expiration time for the cache, and automatically query the database after the cache expires to ensure the consistency of the database and the cache.

If we do not set the expiration time, we must first choose the correct update strategy: update the database and then delete the cache. However, there may be some problems when we delete the cache, so we need to put the cached key to be deleted in the message queue and try again and again until the deletion is successful. 4.2 cache avalanche problem

Problem: we should all have seen an avalanche in the movie, which is calm at first, and then begins to collapse in an instant, which is very destructive. The same is true here. When we execute the code, we set the actual time of many caches to be the same, and then these caches will be effective at the same time, and then all will revisit the database to update the data. this will lead to too many database connections, too much pressure and crash.

Resolve:

Add a random value when setting the cache expiration time. Set double cache, cache 1 sets cache time, cache 2 is not set, 1 returns to cache 2 directly after expiration, and starts a process to update cache 1 and 2. 4.3 cache penetration issu

Problem: cache traversal means that some abnormal users (* *) deliberately request data that does not exist in the cache, causing all requests to be concentrated on the database, resulting in abnormal database connection.

Resolve:

Using mutexes. When the cache fails, the database cannot be accessed directly, but the lock must be acquired before the database can be requested. If you don't get the lock, sleep for a while and try again.

The asynchronous update strategy is adopted. Whether or not the key gets a value, it is returned directly. A cache expiration time is maintained in the value value. If the cache expires, a thread is asynchronously set up to read the database and update the cache. Need to do cache warm-up (load the cache before the project starts).

Provide an interception mechanism that can quickly determine whether the request is valid or not. For example, the Bloom filter is used to maintain a series of legal and valid key internally to quickly determine whether the Key carried by the request is valid or not. If it is illegal, return directly. 4.4 concurrency contention of caches

Question:

The problem of cache concurrency competition mainly occurs when multithreading set a key, when data inconsistency occurs.

For example, in Redis, we store a value of key as amount, and its value is 100. both threads add 100 to value at the same time and then update it, and the correct result should be 300. But when both threads get this value, it is 100, and the final result is 200, which leads to the problem of cache concurrency competition.

Solve

If there is no order requirement for multithreaded operations, we can set a distributed lock, and then multiple threads compete for the lock, and the one who grabs the lock can execute first. This distributed lock can be implemented using zookeeper or Redis itself. You can take advantage of Redis's incr command. When our multithreaded operations need to be sequenced, we can set up a message queue, add the required operations to the message queue, and execute commands strictly according to the sequence of the queue. Fifth, the expiration policy of Redis

Redis as the data increases, the memory usage will continue to increase. We thought that some keys will be deleted when they reach the set deletion time, but when the time is up, the memory occupancy rate is still very high. Why?

Redis uses a memory elimination mechanism of periodic deletion and lazy deletion.

5.1 periodically delete

There is a difference between regular deletion and regular deletion:

Scheduled deletion must strictly follow the set time to delete the cache, which requires us to set a timer to constantly poll all key to determine whether it needs to be deleted. But in this way, the resources of cpu will be greatly occupied, and the utilization of resources will become lower. So we choose to delete on a regular basis.

The time for regular deletion is up to us. We can check every other 100ms, but we still can't check all the caches. Redis will still jam and can only randomly check some caches, but some caches cannot be deleted within the specified time. At this point, lazy deletion comes in handy. 5.2 lazy deletion

To give a simple example: in middle school, there was too much homework to finish at all. The teacher said he would talk about this paper in the next class. Have you all finished it? As a matter of fact, there are a lot of people who haven't finished, so they need to make up before the next class.

The same is true of lazy deletion. Our value is supposed to be gone, but it is still there. When you want to get this key, you find that the key should have expired, delete it quickly, and return a'no this value, it has expired!'.

Now that we have the expiration policy of regular deletion + lazy deletion, can we rest easy? This is not the case. If this key is not accessed all the time, it will be stuck all the time, which is unreasonable, which requires our memory obsolescence mechanism.

5.3memory obsolescence mechanism for Redis

There are generally 6 memory obsolescence mechanisms for Redis, as shown in the following figure:

So how do we configure Redis's memory obsolescence mechanism?

We can configure it in Redis.conf

# maxmemory-policy allkeys- LRU VI. Summary

In this paper, we explore Redis and sort out the knowledge graph of Redis. By comparison, we can find that Redis has so many knowledge points to learn. Then we analyze the advantages and disadvantages of Redis, know its efficient read and write speed based on memory and rich data types, and analyze how Redis should deal with the problems such as data consistency, cache penetration, cache avalanche and so on. Finally, we learned about the expiration strategy and cache elimination mechanism of Redis.

I believe you already have some understanding of Redis, and in the next article we will analyze the data structure of Redis, how each data type is implemented, and what the corresponding commands are.

Author: Yang Heng

Source: Yixin Institute of Technology

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.