Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to quickly troubleshoot and find the bigkey of redis". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to quickly troubleshoot and find the bigkey of redis".

Let's meet redis and bigkey first.

The darling of redis-- Internet

Redis, as an excellent industrial-grade memory database, has gradually become the darling of the Internet since its birth, supporting the rich and colorful functions of the Internet and huge QPS (query rate per second), and has become synonymous with high performance like nginx. For example, there is a redis waiting behind every hot search on Weibo. In a sense, redis usage also represents the traffic of an Internet company.

In the von Neumann computing system, memory is an important existence. There is a certain dialectical relationship between computing and storage. Storage can be reduced through calculation, and a certain amount of consumption can be reduced through storage. Therefore, the idea of cache has also become an important optimization means to reduce the amount of computation in system performance.

Let's look at two comparative pictures.

This is a performance comparison of cpu, memory, and disk. The read-write performance of memory is nearly a thousand times that of disk. As a memory-based storage medium, redis has a revolutionary change to improve system performance, and the economic base determines the superstructure, so economic benefits are always the first.

Let's take a look at the prices of all kinds of storage.

Another design implementation that underpins the high performance of redis is single-threaded task processing design.

In the face of a huge workload, in order to complete the task as soon as possible, we usually choose to add people and disassemble the task into multiple parallel processing. This is a simple idea of multithreaded concurrent processing, which can improve the throughput of task processing.

So why does redis do the opposite and use a single-threaded approach? The single thread here means that Redis's network IO and key-value pairs are read and written by a single thread, which is also the main process for Redis to provide key-value storage services.

First, if you use multithreading, what are the problems when dealing with network IO? Thread is the basic unit of cpu scheduling. Cpu switches time slices through clock interruptions, showing the effect of parallel processing, and task switching is bound to consume resources. When a large number of threads are opened up to a certain amount, these switches will reach the performance bottleneck.

There is also the problem of resource sharing under multithreading, which is also the core issue of concurrent programming. If there is resource competition in the concurrent environment, it is necessary to lock the critical area of shared resources and transform the concurrent processing into serialization. In the data structure of redis, the hashMap index data structure used at the bottom will inevitably lead to the problem of resource competition in the case of multi-thread processing, which becomes a serial synchronization process again. Therefore, we negate multithreading here.

So let's take a look at the benefits of single threading in this scenario.

After the three-way handshake between the client and the server through tcp, the network connection is established, and the request data is written from the network card to the kernel buffer of the operating system. When the user program performs the read operation, the data will be written from the kernel space to the variables executed by the user program. If the data has not been received in the kernel space, blocking waiting will occur here.

This kind of treatment is obviously unacceptable to us. In order to solve this technical problem, a technology called IO multiplexing has been created. There are three implementations under linux. Select, poll, and epoll, simply put, this mechanism allows multiple listening sockets and connected sockets to exist in the kernel at the same time, and then read and write is ready to inform the user to execute. Specific here do not expand, interested friends can search the Internet.

The gateway layer technologies such as nodejs and nginx are designed with single thread. In dealing with network IO, single thread is better than multi-thread. However, single threading also has its fatal weakness, once one of the request tasks in processing is too long, it will block subsequent requests. This is also the main reason for the harm of bigkey to be carried out later. Just like on an one-way street, when a car breaks down, there will be a traffic jam.

What is bigkey?

From the underlying data storage structure of redis in the figure above, you can see that value has the implementation of multiple data structures, so the size of value is represented by the length of a string, while the size of value is represented by the number of elements when it is a compound type.

Bigkey is the big value problem in redis key/value system. According to the classification of data types, bigkey is reflected in two points:

The stored data is of string type, and the length of value value is too large.

Value is a compound type and contains too many elements.

In redis, a string has the largest 512MB, and a secondary data structure (such as hash, list, set, zset) can store about 4 billion (2 ^ 32-1) elements, which is a theoretical value. In practice, we can comprehensively measure the limit number through the data given by operation and maintenance. Generally, the number of string types is controlled within 10KB, and the number of compound types hash, list, set, and zset elements does not exceed 5000.

Second, what is the harm of bigkey? Cause?

Seeing here, we already have a preliminary understanding of bigkey. Next, we introduce the harm and causes of bigkey one by one.

1. Four major hazards of bigkey

As the saying goes, "one mouse shit spoils a pot of soup". To redis, bigkey is like the existence of rat shit. The danger is mainly manifested in the following four aspects:

1. Uneven memory space

In the cluster mode, due to the existence of bigkey, the memory of the host node is uneven, which is not conducive to the unified management of memory in the cluster, and there is a hidden danger of data loss.

two。 Timeout blocking

Because of the single-threaded nature of redis, it is usually time-consuming to operate bigkey, which means that the more likely it is to block redis, which can cause client blocking or cause failover. They are usually found in slow queries.

3. Network congestion

Bigkey also means that each acquisition will generate a large amount of network traffic. Suppose that a bigkey is 1MB and the client has 1000 visits per second, then the traffic generated by 1000MB per second is simply a disaster for a server with an ordinary gigabit network card (128MB/s by byte).

4. Blocking deletion

There is a bigkey that sets the expiration time and will be deleted when it expires. If you use a version prior to Redis 4.0and the expired key is deleted asynchronously, it is possible to block the redis, and the expired deletion will not be found in the slow query (because the deletion is not caused by the client, but an internal loop event).

2. How did bigkey come into being?

The emergence of bigkey is mainly due to improper program design, such as the following common business scenarios

Social category: fan list, if some stars or big v is not carefully designed, it must be bigkey.

Statistical class: for example, storing a function or a collection of users of a website on a daily basis must be bigkey unless few people use it.

Cache class: serialize data from the database load to redis, which is often used, but there are two things to pay attention to: first, whether it is necessary to cache all fields; second, whether there is related data.

Thus, in the program design, we should have a basic assessment of the growth and boundary of the amount of data, and do a good job in technology selection and technology architecture.

Three or four solutions to bigkey found by investigation

Let's take a look at a thought question:

At the beginning of today, the COVID-19 epidemic broke out one after another in Shijiazhuang. For a large and medium-sized city with a population of more than 10 million, the prevention and control of the epidemic is facing tremendous pressure. How to efficiently detect people infected by the virus and people in contact with people has become the key to the prevention and control of the epidemic. The Government has done the following work, which can be summarized into four points:

1. The circulation of personnel is prohibited and the home is isolated.

two。 Establish a risk level

3. Gridding management

4. Accounting and testing.

According to the characteristics of epidemiological medicine, symptoms must be reported actively. This is an active report, because the COVID-19 epidemic has a certain incubation period, and many asymptomatic patients, so we need to actively find out through the mechanism of accounting and testing that this actually embodies the idea of computer processing and scanning.

It is found that the idea of dealing with bigkey is similar to the practice of epidemic prevention and control, and there are four conventional practices.

1. Redis client tools

Redis-cli provides-- bigkeys to find bigkey. For example, the following is the result of an execution.

Because the execution efficiency of debug object key is very slow, it is possible to block the Redis thread. Therefore, this scheme will also do some damage to the business, and when in use, the executive program can be run on the slave node.

3. RDB file scanning

We know that redis has a persistence scheme called RDB persistence, which is a disk snapshot of redis data stored in memory. You can scan the RDB file through the RDB tool to find out the existence of bigkey.

When choosing this option, you first need to persist the RDB file. RDB persistence is a form of memory snapshots. Snapshots are taken down according to a certain frequency. This solution is an ideal choice and will not affect the operation of redis hosts. However, in scenarios that require high data reliability, RDB persistence scheme will not be selected, so it is not universally applicable.

4. The scanning design idea of DataFlux bigkey.

The previous schemes are either discovered by the client or need to scan all the data, which is a behavior that consumes computing resources. The analogy of the epidemic situation is like testing the nucleic acid of all the staff regardless of grade after the emergence of a case in a city with a population of 10 million. This not only consumes huge material resources, financial resources, and manpower, but also is very inefficient. It runs counter to the epidemic prevention and control that needs to race against time.

We have also analyzed the causes of redis bigkey above, many of which are caused by unreasonable business design and inadequate evaluation. Therefore, in the design of DataFlux products, datakit's redis collector uses a scheme to independently configure potential bigkey for scanning and discovery, which supports fixed key values and key pattern. In key pattern, a certain range of key is obtained through scan pattern, and then the value of each type of key ("HLEN", "LLEN", "SCARD", "ZCARD", "PFCOUNT"STRLEN") is taken by the length function, and the corresponding length of key is obtained, which is reported to the DataFlux platform for monitoring and storage.

The advantages of this approach are as follows:

First, because the length is obtained for the target key, and the acquisition of the duration value of various data types in redis is O (1) time complexity. Therefore, the execution efficiency is very high.

Second, the collected result data is reported to the DataFlux storage platform, where you can display all kinds of charts of the index data and monitor the alarm.

Simulate the initial queue push 10 value

Then push a certain amount of data

On the DataFlux platform, through the indicators, the monitored key can be displayed by chart, monitoring alarm, visual display and so on, to maximize the value of the data.

Thank you for reading, the above is the content of "how to quickly troubleshoot and find the bigkey of redis". After the study of this article, I believe you have a deeper understanding of how to quickly troubleshoot and find the bigkey of redis, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.