Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the basic concept of Storm". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the basic concept of Storm".

1. Basic concepts of Storm:

Nimbus: responsible for resource allocation and task scheduling

Supervisor: responsible for the tasks assigned by Nimbus, starting and stopping worker processes managed by yourself

Worker: a component process that runs specific processing logic

Spout/Bolt (Task): each Spout/Bolt thread in worker is called a Task. After the storm0.8 version, the task no longer corresponds to the physical thread. The Task of the same Spout/Bolt may share a physical thread, which is called executor.

Topology: the arrangement and connection of all components (that is, Spouts/Bolts) is called a Topology

Spout: the component that generates the source data stream

Bolt: components that receive and execute data

Tuple: the basic unit of message delivery (transporting messages in Tuple)

Stream: the continuous flow of Tuple makes up Stream.

The process of data processing:

Processing is assigned to different types of components (components), each responsible for a simple, specific processing task. The component that processes the input stream of the Storm cluster is called the spout, which then passes the data to a component called bolt and processes the data in the bolt. When the processing is complete, the bolt either stores the data (in a database, disk, or even an object) or passes it on to other bolts. Therefore, you can think of a Storm cluster as a chain of bolts, each of which does some processing to the data sent by the nozzle.

2. Storm features (advantages / benefits)

Simple programming, multi-language programming, fault-tolerant, extensible, reliable, fast, transactional

Difference from other real-time computing systems (S4, puma)

Record-level fault tolerance: tells the user whether each message unit is fully processed within a specified time (whether the message unit is fully processed by each Bolt component of Topology)

1. Spout specifies a MessageId for Tuple when sending a new source (this MessageId can be any object)

2. There is a system-level component in Topology called acker,acker to track the processing paths of several Tuple flows from binding MessageId in Spout.

3. If the tuple is not fully processed within the maximum timeout set by the user, acker will inform spout that the message processing has failed. On the contrary, it will inform spout that the message processing was successful.

Transaction topology (Transactional Topology): the new feature introduced in storm0.7 version is replaced by Trident in version 0.8 and deprecated in version 0.9 to meet scenarios with extremely stringent requirements for message processing, such as real-time counting of a user's transaction number, requiring complete accuracy, neither more nor less.

Thank you for your reading, the above is the content of "what is the basic concept of Storm", after the study of this article, I believe you have a deeper understanding of what the basic concept of Storm is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.