Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the ways to cancel the storm tracking mechanism". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what are the methods to cancel the storm tracking mechanism"?

Nouns involved in storm

0 、 zookeeper

1 、 nimbus

2 、 supervisor

3 、 worker

4 、 executor

5 、 task

6 、 topology

0. Zookeeper: store status information, scheduling information, heartbeat,-"the communication between nimbus and supervisor is connected by zookeeper, nimbus and supervisor cannot communicate directly, and the information in the storm cluster is stored in zookeeper.

1. Nimbus: equivalent to the master,storm in the cluster is the master/slave architecture-"nimbus has two main tasks: 1 nimbus -" manages the cluster, supervisor writes information to zookeeper, and then nimbus gets the information from the nodes of the zookeeper, so that nimbus knows the information in the cluster.

2Murray-"dispatch topology. When a topology is submitted to the cluster through nimbus, nimbus will get the number of supervisor and health status in the cluster through zookeeper, and then launch worker,nimbus to receive data according to the health status of supervisor.

2. There will be a supervisor process on each node of supervisor:storm, and supervisor is responsible for starting and stopping the worker process.

3, worker: each supervisor process generally starts multiple worker processes, worker is jvm, mainly responsible for two things; 1, start executor,2, responsible for the communication between worker and worker.

4. Executor: create the actual Spout/Bolt object, create two threads, execute thread and transport thread (responsible for the communication between task and data in task).

5. Task: I think task is the thread of execution in executor.

6. Topology: it's the storm program we wrote.

7. Spout (faucet): every topology must have a Spout node, which must start from the Spout node, which is the source of production Tuple. Spout has KafkaSpout, DRPCSpout, RedisSpout, etc. Spout is the starting node of the cluster, so nimbus and supervisor are down, and the cluster can still operate. Spout takes the data and then generates Tuple, and then sends the data to Bolt to process the data. NextTuple () is the function of Spout to send data, and emit () is the method of sending data.

8. Bolt: each topology can have n Bolt processing nodes to deal with the intermediate state or result state of the data. The execute () method in Bolt is used to process the data.

BaseBolt:

RichBolt:collector.emit (oldTuple,newTuple)

9. Tuple:Tuple 's id is very useful for the reliability of Storm, the smallest unit of data

10. Acker:Storm tracking mechanism, Tuple will report to acker at each level of processing; if the required information is not returned within a specified time, there will be a certain processing mechanism, such as resending or discarding

Three methods of canceling tracking Mechanism

1. If Config.TOPOLOGY_ACKERS is set to 0 in the configuration file, the whole bolt tree structure has no ack mechanism.

2. Without setting messageid, XOR cannot be done without messageid,ack.

3. Do not send oldTuple when sending Tuple

Tracking summary: the implementation of the ack mechanism needs to be configured first, followed by messageid, and finally, both the original Tuple and the new Tuple should be sent out.

10. Stream: every Stream has an id. If it is not specified to write id, the default stream is used, called default, and each Spout/Bolt has a default Stream:default.

Note: concurrency is measured by how many executor (thread), not by task

Thank you for your reading, the above is the content of "what are the methods to cancel the storm tracking mechanism". After the study of this article, I believe you have a deeper understanding of what the method of canceling the storm tracking mechanism has, and the specific use still needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.