Shulou(Shulou.com)06/03 Report--

I. the background of streaming computing

In daily life, we usually store the data in a table first, and then process and analyze it, which involves a timeliness problem. If we deal with data at the level of year and month, the real-time requirement of much data is not high, but if we are dealing with data in days, hours, or even minutes, then the requirement for timeliness of the data is relatively high. In the second scenario, if we still use the traditional data processing method, uniformly collect data, store it in the database, and then analyze it, we may not be able to meet the requirements of timeliness.

II. Flow calculation and batch calculation

Big data's computing model is mainly divided into batch computing (batch computing), streaming computing (stream computing), interactive computing (interactive computing), graphical computing (graph computing) and so on. Among them, streaming computing and batch computing are the two main big data computing modes, which are suitable for different big data application scenarios.

Stream data (or data stream) refers to an infinite series of dynamic data sets in time distribution and quantity. The value of data decreases with the passage of time, so it is necessary to calculate and give a second-order response in real time. Streaming computing, as the name implies, is the processing of data flow, is real-time computing. Batch computing is the data calculation method of uniformly collecting data, storing it in the database, and then batch processing the data. It is mainly reflected in the following aspects:

1. The timeliness of data is different: real-time streaming computing, low latency, non-real-time batch computing, high latency.

2. The data characteristics are different: the streaming data is generally dynamic and boundless, while the batch data is generally static data.

3. Different application scenarios: streaming computing is used in real-time scenarios with high timeliness requirements, such as real-time recommendation and business monitoring. Batch computing is generally said to batch processing, applied in real-time requirements is not high, offline computing scenarios, data analysis, offline reports and so on.

4. The operation mode is different, the task of streaming computing is continuous, and the task of batch computing is completed at one time.

III. Streaming computing framework, platform and related products

First, commercial streaming computing platforms (IBM InfoSphere Streams, IBM StreamBase, etc.)

The second category, open source flow computing framework (Twitter Storm, S4, etc.)

The third category is the streaming computing framework developed by the company to support its own business.

The first generation stream processing system developed by Strom:Twitter.

The second generation stream processing system developed by Heron:Twitter.

Spark streaming: an extension of Spark core API, it can achieve high-throughput, fault-tolerant real-time streaming data processing.

Flink: is a distributed processing engine for streaming and batch data.

Apache Kafka: written by Scala. The goal of the project is to provide a unified, high-throughput, low-wait platform for processing real-time data.

IV. Main application scenarios of streaming computing

Streaming can be used in two different scenarios: event flow and continuous computing.

1. Event flow

Event flow tools can continuously generate a large amount of data, which first appeared in the traditional banking and stock trading fields. it also appears in Internet monitoring, wireless communication networks and other fields, which need to analyze the updated data flow in near real-time, such as trend analysis, prediction, monitoring and so on. To put it simply, the event flow uses a way in which queries remain static, statements are fixed, and data is constantly changing.

2. Continuous calculation

For example, for the streaming data of large websites: website visit PV/UV, what content users visit, what content they search, and so on, real-time data calculation and analysis can dynamically refresh user access data in real time, show the changes of real-time traffic of the website, and analyze the traffic and user distribution of each hour of the day.

In the financial sector, for example, the need for millisecond delay is crucial. Storm can also be used in some scenarios that need to process data in real time, such as real-time analysis of log files generated by user behavior, real-time recommendation of products to users, and so on.

V. the value of flow calculation

We get the value of the data through big data's processing, but is the value of the data constant? Obviously not, some data have a higher value soon after the event, and that value will decrease rapidly over time. The key advantage of streaming is that it provides insight faster, usually between milliseconds and seconds.

The value of streaming computing is that businesses can mine the value of business data in a shorter time and turn this low latency into a competitive advantage. For example, in recommendation engines using streaming computing, users' behavior preferences can be reflected in the recommendation model in a shorter time. The recommendation model can capture users' behavior preferences with lower latency to provide more accurate and timely recommendations.

The reason why streaming computing can do this is that traditional batch computing needs data accumulation and batch processing after accumulating a certain amount of data, while streaming computing can achieve data processing as soon as possible, effectively reducing the processing delay.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.