Shulou(Shulou.com)06/01 Report--

This article focuses on "what is the basic architecture of Storm". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the basic architecture of Storm"?

At present, the commonly used streaming real-time computing engines are divided into two categories: line-oriented and micro-batch-oriented. The representative of the line-oriented streaming real-time computing engine is Apache Storm, which is typically characterized by low latency but low throughput. The representative of the streaming real-time computing engine for micro-batch processing is Spark Streaming, which is typically characterized by high latency but high throughput.

Mainstream streaming data lines are divided into four phases:

1. Data acquisition: responsible for real-time data collection from different data sources, including Flume and custom Kafka Producer

2. Data buffering: in order to balance the data acquisition rate and the data processing rate is not equal. Eg:Kafka

3. Real-time analysis: get the data from the data buffer and finish the data processing quickly. For example: Storm and Spark Streaming

4, result storage: store the calculation results to external systems, such as: a large number of real-time query systems, can store Hbase, a small amount but need a highly concurrent query system, can store Redis.

Storm:

Basic concepts:

1. Tuple: consists of a set of serializable elements.

2. Stream: infinite Tuple forms a Stream

3. Topology: job similar to MapReduce, a DAG consisting of a series of Spout and Blot

4. The data source of Spout:Stream

5. Bolt: message processing logic

Basic architecture:

1. Nimbus: cluster management and scheduling component

2. Supervisor: computing component

3. Coordination components before Zookeeper:Nimbus and Supervisor.

Spark Streaming:

Basic concept: the core idea is to convert streaming processing into "micro-batch processing", that is, the data stream is split in time, and the data in each slice corresponds to a RDD, and then the Spark engine is used for fast calculation. It is precisely because of the micro-batch processing mode that Spark Streaming can only be used as a near real-time processing system, rather than a strict real-time streaming processing.

Spark Streaming further abstracts the streaming data, it batches the streaming data, and each batch of data is abstracted into RDD, so the streaming data becomes a streaming RDD sequence, that is, Dstream,Spark Streaming defines a series of operations on Dstream, which are mainly divided into two categories: transformation and output, in which a transformation operation can transform a Dstream into another Dstream, while an output operation can produce one or a set of results and output them to a designated external system.

Abstraction and operation comparison of several components of Spark:

1. Spark Code data abstraction RDD data operation transformation&action

2. Spark SQL data abstraction: DataFrame, DataSet, operation: transformation&action

3. Spark Streaming: data abstraction: Dstream, operation: transformation&action

At this point, I believe you have a deeper understanding of "what is the basic architecture of Storm". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.