Shulou(Shulou.com)06/01 Report--

Redis hot spot Key discovery and common solutions, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

First, the causes of hot Key problems. 1. The data of user consumption is much larger than that of production (hot-selling goods, hot news, hot comments, star live broadcast).

In some unexpected events in daily work and life, such as the price reduction and promotion of some popular goods during the Singles' Day holiday, when one of the items is viewed or purchased tens of thousands of times, a large demand will be formed. in this case, it will cause hot issues.

By the same token, hot news, hot comments, live broadcast by stars, etc., which are published and browsed in large numbers, these typical scenes of reading more and writing less will also lead to hot issues.

2. Request sharding set, which exceeds the performance limit of a single Server.

When the server reads data for access, the data is often sliced and segmented. In this process, the corresponding Key is accessed on a host Server. When the access exceeds the Server limit, it will lead to hot Key problems.

II. The harm of hot Key issues

1. The traffic is concentrated, reaching the upper limit of the physical Nic.

2. If there are too many requests, the cache sharding service is destroyed.

3. DB breakdown, causing business avalanche.

As mentioned earlier, when the request of a hot Key exceeds the upper limit of the host network card on a certain host, due to the excessive concentration of traffic, other services in the server will be unable to carry out.

If the hot spot is too concentrated and the cache of the hot spot Key is too much, it will lead to the collapse of the cache sharding service when it exceeds the current cache capacity.

When the cache service crashes, another request is generated and cached to the backend DB. Due to the weak performance of DB itself, request penetration is easy to occur in the face of large requests, which will further lead to avalanche phenomenon and seriously affect the performance of the device.

Third, the usual solution is mainly focused on the corresponding transformation of the client and the Server side. 1. Server cache scheme

First, Client sends the request to Server, and Server is a multi-threaded service, so there is a cache space based on Cache LRU policy locally.

When the Server itself is congested, Server does not send the request further to the DB but returns directly, only when the Server itself is unblocked will the Client request be sent to the DB and the data will be rewritten to the cache.

At this point, the access and reconstruction of the cache are completed.

However, the scheme also has the following problems:

Cache invalidation, multi-thread build cache problem cache loss, cache construction problem dirty reading problem 2, use Memcache, Redis scheme

This scheme solves the hot Key problem by deploying cache separately on the client side.

In the process of use, Client first visits the service layer, and then accesses the cache layer on the same host.

This solution has the advantages of nearest access, high speed, and no bandwidth restrictions, but it also has the following problems:

Waste of memory resources dirty read problem 3. Use local cache scheme

Using local caching has the following problems:

Need to know in advance that hot spot cache capacity is limited, inconsistent time growth hot spot Key omitted

There are all kinds of problems in traditional hot spot solutions, so how to solve them?

4. Read-write separation solution for hot reading

The roles of each node in the architecture are as follows:

SLB layer does load balancing Proxy layer does read-write separation automatic routing Master is responsible for writing requests ReadOnly nodes are responsible for reading requests Slave nodes and Master nodes are highly available

In the actual process, Client sends the request to SLB,SLB and distributes it to multiple Proxy, and then classifies and sends the request through the identification of Proxy.

For example, a request that is also a Write is sent to the Master module, and a request for Read is sent to the ReadOnly module.

The read-only nodes in the module can be further expanded to effectively solve the problem of hot reading.

At the same time, read-write separation has the advantages of flexible expansion of read hot spots, storage of a large number of hot Key, client-friendly and so on.

5. Hot data solution

This scheme solves the problem of hot Key by actively discovering hot spots and storing them.

First, Client also accesses SLB and distributes various requests to Proxy through SLB, and Proxy forwards the request to the back-end Redis in a route-based manner.

The hot key is solved by adding cache on the server side.

Specifically, the local cache is added on the Proxy, the local cache uses the LRU algorithm to cache the hot spot data, and the back-end db node adds the hot spot data calculation module to return the hot spot data.

The main advantages of Proxy architecture are as follows:

Proxy local cache hotspot, read capacity can be horizontally expanded DB node timing calculation hotspot data collection DB feedback Proxy hotspot data is completely transparent to the client, there is no need to do any compatible hot spot key processing 1, hot spot data reading

The processing of hot Key is mainly divided into two forms: writing and reading. In the process of data writing, when SLB receives data K1 and writes it to a Redis through a certain Proxy, it completes the data writing.

If K1 becomes a hotspot key after the calculation of the backend hotspot module, Proxy will cache the hotspot, and the next time the client accesses K1, it can not go through Redis.

Finally, because proxy can be expanded horizontally, the access ability of hot spot data can be enhanced arbitrarily.

2. Discovery of hot spot data

For the discovery of hot spot data on db, first of all, the request statistics of Key will be carried out in a cycle. After reaching the request level, the hot spot Key will be located, and all hot spot Key will be put into a small LRU linked list. When accessing through Proxy request, if Redis finds that the point to be visited is a hot spot, it will enter a feedback stage and mark the data at the same time.

When DB computing is hot, the main methods and advantages are:

1. Hot spot statistics based on statistical threshold 2, hot spot statistics based on statistical period 3, statistical method without resetting initial value based on version number 4. DB calculation also has advantages such as minimal impact on performance and minimal memory footprint. Fifth, comparison of schemes.

Through the above comparative analysis, we can see that there is a great improvement in solving hot Key compared with traditional methods. No matter it is based on read-write separation scheme or hot data solution, it can be flexibly expanded in the actual processing environment, transparent to the client, and has certain data inconsistency.

In addition, the read-write separation mode can store more hot data, while the Proxy-based mode has a cost advantage.

After reading the above, have you mastered the methods of Redis hot spot Key discovery and common solutions? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.