Shulou(Shulou.com)05/31 Report--

This article mainly introduces how the hot Key in Redis is produced, which has a certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article. Let's take a look at it.

The cause of Hot spot Key

1. The data of user consumption is much larger than that of production.

The hot key problem is that there are a large number of requests to access a fixed key on the redis in an instant, resulting in cache breakdown and requests hitting the DB, which overwhelms the cache service and DB service, thus affecting the availability of application services.

The most common are hot searches on Weibo, such as XX stars getting married / having an affair. Then the Key of XX stars will increase instantly, and there will be hot data problems. Weibo also collapses from time to time.

By the same token, hot news, hot comments, live broadcast by stars, etc., which are published and browsed in large numbers, these typical scenes of reading more and writing less will also lead to hot issues.

2. Request sharding set, which exceeds the performance limit of a single Server

When the server reads data for access, the data is often sliced and segmented. In this process, the corresponding Key is accessed on a host Server. When the access exceeds the Server limit, it will lead to hot Key problems.

The harm of Hot Key issues

1. The traffic is concentrated, reaching the upper limit of the physical Nic.

When the request of a hotspot Key exceeds the upper limit of the host network card on a certain host, due to the excessive concentration of traffic, other services in the server can not be carried out.

2. If there are too many requests, the cache sharding service is destroyed.

If the hot spot is too concentrated and the cache of the hot spot Key is too much, it will lead to the collapse of the cache sharding service when it exceeds the current cache capacity.

3. DB breakdown, causing business avalanche.

When the cache service crashes, another request is generated and cached to the backend DB. Due to the weak performance of DB itself, request penetration is easy to occur in the face of large requests, which will further lead to avalanche phenomenon and seriously affect the performance of the device.

The discovery of hot spot key

1. Estimate which hot key is based on business experience.

In fact, this method is quite feasible. For example, if a product is doing a second kill, then the key of this product can be judged to be hot key. The drawback is that not all businesses can predict which key is hot key.

2. Collect on the client

This way is to add a line of code for data statistics before operating redis. Well, there are many ways to count this data, and it can also be to send a notification message to an external communication system. The disadvantage is that it invades the client code.

3. Collect at the Proxy layer

# bytemd-mermaid-1632312967851-0 {font-family: "trebuchet ms", verdana,arial,sans-serif;font-size:16px;fill:#333;} # bytemd-mermaid-1632312967851-0. Error-icon {fill:#552222;} # bytemd-mermaid-1632312967851-0. Error-text {fill:#552222;stroke:#552222;} # bytemd-mermaid-1632312967851-0.edge-thickness-normal {stroke-width:2px;} # bytemd-mermaid-1632312967851-0.edge-thickness-thick {stroke-width:3.5px } # bytemd-mermaid-1632312967851-0 .edge-pattern-solid {stroke-dasharray:0;} # bytemd-mermaid-1632312967851-0 .edge-pattern-dashed {stroke-dasharray:3;} # bytemd-mermaid-1632312967851-0 .edge-pattern-dotted {stroke-dasharray:2;} # bytemd-mermaid-1632312967851-0 .questions {fill:#333333;stroke:#333333;} # bytemd-mermaid-1632312967851-0 .marker.cross {stroke:#333333 } # bytemd-mermaid-1632312967851-0 svg {font-family: "trebuchet ms", verdana,arial,sans-serif;font-size:16px;} # bytemd-mermaid-1632312967851-0 .label {font-family: "trebuchet ms", verdana,arial,sans-serif;color:#333;} # bytemd-mermaid-1632312967851-0. Cluster-label text {fill:#333;} # bytemd-mermaid-1632312967851-0. Cluster-label span {color:#333 } # bytemd-mermaid-1632312967851-0. Label text,#bytemd-mermaid-1632312967851-0 span {fill:#333;color:#333;} # bytemd-mermaid-1632312967851-0. Node rect,#bytemd-mermaid-1632312967851-0. Node circle,#bytemd-mermaid-1632312967851-0. Node ellipse,#bytemd-mermaid-1632312967851-0. Node polygon,#bytemd-mermaid-1632312967851-0. Node path {fill:#ECECFF;stroke:#9370DB;stroke-width:1px;} # bytemd-mermaid-1632312967851-0. Node. Label {text-align:center } # bytemd-mermaid-1632312967851-0 .node.clickable {cursor:pointer;} # bytemd-mermaid-1632312967851-0 .arrowheadPath {fill:#333333;} # bytemd-mermaid-1632312967851-0 .edgePath .path {stroke:#333333;stroke-width:1.5px;} # bytemd-mermaid-1632312967851-0. Flowchart-link {stroke:#333333;fill:none;} # bytemd-mermaid-1632312967851-0 .edgeLabel {background-color:#e8e8e8;text-align:center;} # bytemd-mermaid-1632312967851-0 .edgeLabel rect {opacity:0.5 Background-color:#e8e8e8;fill:#e8e8e8;} # bytemd-mermaid-1632312967851-0 .cluster rect {fill:#ffffde;stroke:#aaaa33;stroke-width:1px;} # bytemd-mermaid-1632312967851-0 .cluster text {fill:#333;} # bytemd-mermaid-1632312967851-0 .cluster span {color:#333;} # bytemd-mermaid-1632312967851-0 div.mermaidTooltip {position:absolute;text-align:center;max-width:200px;padding:2px;font-family: "trebuchet ms", verdana,arial,sans-serif Font-size:12px;background:hsl (80100% bytemd-mermaid-1632312967851 96.2745098039%); border:1px solid # aaaa33;border-radius:2px;pointer-events:none;z-index:100;} # bytemd-mermaid-1632312967851-0:root {--mermaid-font-family: "trebuchet ms", verdana,arial,sans-serif;} # bytemd-mermaid-1632312967851-0 flowchart {fill:apa;}

Client

Proxy

Redis1

Redis2

Redis3

Some cluster architectures are as follows. Proxy can be Twemproxy, which is a unified entry. Collection and reporting can be done at the Proxy layer, but the disadvantage is obvious that not all redis cluster architectures have proxy.

4. Use redis to bring your own commands

Monitor command: this command can grab the commands received by the redis server in real time, and then write code to figure out what the hot key is. Of course, there are ready-made analysis tools for you to use, such as redis-faina. However, under the condition of high concurrency, this command has the hidden danger of memory explosion, and will also degrade the performance of redis.

Hotkeys parameter: redis 4.0.3 provides the hotspot key discovery function of redis-cli, and you can add the-hotkeys option when executing redis-cli. However, when this parameter is executed, if there are more key, it is slower to execute.

5. Grab the bag and evaluate it yourself.

The Redis client uses the TCP protocol to interact with the server, and the communication protocol uses RESP. Write your own program to listen to the port, parse and analyze the data according to the rules of RESP protocol. The disadvantages are high development cost, difficult maintenance and the possibility of packet loss.

Each of the above five schemes has its own advantages and disadvantages. You can make a choice according to your business scenario.

The solution of Hot Key

1. Make use of secondary cache

For example, you can use ehcache, spring cache, or even a HashMap. After you find the hot key, load the hot key into the JVM of the system.

For this hot key request, it is taken directly from the jvm rather than going to the redis layer.

Suppose there are 100, 000 requests for the same key at this time. If there is no local cache, the 100, 000 requests will be directly directed to the same redis.

Now suppose you have 50 machines in your application layer, OK, and you also have jvm cache. The 100, 000 requests are evenly distributed, with 2000 requests per machine, and the value is taken from the JVM and the data is returned. It avoids the situation in which 100, 000 requests go to the same redis.

2. Separation of reading and writing

Read-write separation is to send the request of the same Write to the Master module and the request of Read to the ReadOnly module.

The read-only node in the module can be further expanded to store this key on multiple redis. When a hot key request comes in, we randomly select one on the redis with backup, access the value, and return the data. So as to effectively solve the problem of hot reading.

At the same time, read-write separation has the advantages of flexible expansion of read hot spots, storage of a large number of hot Key, client-friendly and so on.

Thank you for reading this article carefully. I hope the article "how to generate Hot Key in Redis" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.