Shulou(Shulou.com)06/02 Report--

I am very glad to have the opportunity to communicate with you here on Redis5.0 's Stream application. Today's sharing is more of a throwing brick to attract jade. Welcome to put forward more thoughts about Redis.

First of all, let's assume that there is a cup here, which was given by my wife last year because my previous cup was so insulated that I seldom drank water, and such an open cup prompted me to drink more water. Although this cup is only one of thousands of cups on the merchant's shelf, it is still different to me. The difference is the past, it is the memory. This memory is a kind of data, and this kind of data makes our lives better.

What are the characteristics of this kind of data? It happens at once, but we want to see it all the time, and we want to fill it with all kinds of things. The cup itself can also be said to be a production-consumption model: data appears and then consumed.

A situation of consumption

Therefore, a cup is not just a cup, in fact, there are a lot of things to dig behind it. The more meaning, the more connections, the more complex the relationship, the larger the amount of data we have. Therefore, we, who want to maximize the value, produce a large number of data that we want to be processed at a high speed. This data is reflected in the system and often becomes a flood of data. It has become an unbearable burden for the system.

In many cases, the information contained in our collection side may far exceed this value. For example, in haze days, what is our room, our location, what is its air quality, and what are the pollutant parameters? What is the air quality of our office area, this building, this room, what is the power consumption, pedestrian status, vehicle data, and so on. Hundreds of millions of data, involving interconnection, need to ensure high concurrency and reliable transmission. At the same time, after the data is collected, it has to be processed, stored and analyzed, which is a great challenge to the system.

Huge, mesh interconnection requires a huge and smooth pipeline with huge bandwidth; so much data will form a huge data torrent, which can be analyzed in the cloud after collection, which can also produce huge user value.

Area detection and monitoring

Although these data have different forms, they also have something in common, that is, they are related to time. For example, a family, from the master bedroom to the study, from the living room to the dining room, places some air monitors in different rooms. The chemicals in the monitors come into contact with various components in the air and change slowly over time. Signals are generated in the process of change, and these signals are preliminarily collated and calculated to form a plane of air quality data. From morning to evening, from spring to autumn, the values of formaldehyde, TVOC and other pollutants are also different at different time points, so the plane data form the time series data here.

Coincidentally, Redis streams are designed specifically for time series data. Let's review the storage design of Redis's streams: the mainline is a linked list of messages that string all messages in order. Therefore, the support of the Redis stream is very good, and we can add the data in the plane to the Redis stream according to the time series. Of course, we may need preliminary processing of these data, so we can also use other data structures of Redis, such as list, and rely on Redis's support for Lua scripts to drive it to complete processing with very little external application logic. Because this processing is done within Redis, overall, the computational cost is relatively low. Redis originally relies on its native C advantages, as well as memory implementation and data structure optimization, memory footprint is relatively small, CPU requirements are also low, making it possible to run efficiently on small devices. In the future, there may be trillions of smart devices based on the ARM platform, the prospect is still very broad.

So from the device side, we can already use Redis to complete the temporary data storage and basic processing. After joining the Redis stream, we can use MQTT, TCP, 808 and other protocols to upload data through the network. Usually we need to collect a wide area, a large number of equipment, so there is a lot of data, and the data may need to be collected locally. The flood of data often begins to emerge here. If we expect to save it in a database such as mysql, we usually can't stand the pressure. Because of the atomicity, consistency, isolation and persistence of the database, the loss of performance is relatively large. So here we can use Redis to receive flood peak monitoring data, and then distribute it to storage services, processing services, display services, and so on. Before the completion of the distribution process, Redis itself is stored in high-speed memory, and the data in the stream can also be accessed repeatedly as ordinary cached data, so to a certain extent, the gap before message consumption is supplemented, and more processing time is given to the backend.

Let's review the use of Redis: fast, the performance of Redis is very high; small, light and simple, small memory and CPU requirements; rich, rich data structures, a variety of uses; timing, flow is designed for timing applications; support Lua scripts, can customize logic.

Eat the meal one mouthful at a time, and the road has to go step by step. Let's go back to the technology itself, cut off some of the Redis-related parts from the huge torrent and restore them to the basic workflow.

What I am intercepting here is the storage and processing of air monitoring. Let's take a look at the schematic diagram, the instrument reports the data, Redis receives the data stream, and provides it for storage, analysis, consumption, and supply for use.

Test data generation, group establishment

First of all, there is a piece of air data, the detection data is generated, and the consumer group for storage and analysis is also established. The air data contains hcho and tvoc pollution values. Let's take a look at the command, there is a maxlen, this is to avoid the queue is too long, so set the maximum length. Why do you need to set this up? Because after the stream sequence of Redis is consumed, or even after xdel, the data will not be cleaned up, and the queue will become longer and longer, so here we set a maximum length to avoid overflow.

There are two storage services. Assuming that storage is relatively slow, we have built more storage services in order to handle it in a timely manner.

Let's look at the process of storing and analyzing service consumption data. Here both storage services try to get the data, but it is clear that only one gets the data for processing. At this time, the analysis tries to add three pieces of data to the analysis and processing, but because there is only one item in the stream, there is only one item here.

Store and analyze service consumption data

The analysis service takes the lead in completing the data consumption, so the analysis service immediately replies with an ack to stream, telling it that the consumption is complete. The storage service then completes the storage, and the storage service sends an ack to the stream.

Consumption completed, reply to ACK

The analysis service just replied to ack and, for some reason, restarted. When the analysis service starts, it is not clear where the consumption is, so try to start the consumption from the initial location. Here, because ack has been consumed and returned, no consumable data has been fetched. If there is data that is not consumed, it can be consumed again in this way, so the service needs to be able to handle reentry during consumption.

Analysis service is restarted, starting from the beginning of consumption

Here we see that test data is constantly coming, and storage and analysis services are also consumed in accordance with the previous rules. The analysis service still tries to access three items at a time according to its own ability. according to the results, we can see that the analysis service has obtained two pieces of consumption data.

New test data

Analyze service consumption data

What about storage services? Both storage services also fetch data for consumption.

Storage service consumption data

At this point, the user starts to visit the app, and he plans to see what the pollution is like. We all understand that when we just buy a new thing, we are more likely to take a look at it right away, so users here must expect to see pollutants. But at this time, the data is neither analyzed nor stored, what do we need to do with it?

The application service can check the status first and find that the required data has not been processed yet, so it is obviously impossible to get it from the regular cache. So when the app is obtained directly from stream, we can see that users can see the monitoring values smoothly and immediately have an understanding of the situation around them. Users want to see if there is any new data, so click Refresh on the app, and we can see that this time we can still get the data we need from stream. Of course, if users want to take a targeted look at the situation, they can also specify ID reads in the application. Is there any other way? Xrange is also a way to batch fetch required messages.

Applications use xread to read data

Applications use xrange to intercept data

From here, we can feel that although our days are gone forever in the flow of time, fortunately, in the stream of Redis, we can still take any section from the past, taste it again, and review the old and learn the new.

Back to the example. After the user has brushed it twice, the storage and analysis service has processed the data, so the storage and analysis service sends an ack message to stream again, indicating that the processing is complete.

After consumption, return to ack

Of course, the processing in these Redis is as high-performance and efficient as ever.

This is just a simple section, a sign. In fact, the original use of Redis is very rich, for example, as a session cache, as a full-page cache for pages, as a verification code for mobile phones or web pages, frequency restrictions on service access, password anti-brute force cracking, arena, chicken eating, short video goddess list and other rankings, like, reading and other counters and rankings, time-limited discount activities for people who follow a certain tag or a certain star. The real-time index calculation of securities, the number dispatcher, and even geo geographic information, custom logic based on LUA, subscription and publication, etc., are very rich. These are based on the rich data structure of Redis, the development of the use of.

Luo Pang's friend in the time speech said that the general trend is often not a small trend gradually growing up, but the trend impact trend, change brings change, gradually rolling, intertwined and larger. So what kind of collisions can Redis streams provide in these existing application scenarios? What kind of trend can it bring in new areas? Welcome to join us for discussion.

You are also welcome to go to Huawei Cloud distributed Cache to get Redis 5.0for free. Now Redis 5.0is a public trial phase and can be tested for free. It is also very easy to get, apply for a public test, and then take a few seconds to create an instance of Redis 5.0. it will be fine.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.