Shulou(Shulou.com)05/31 Report--

This article focuses on "what is the principle and architecture of Storm". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the principle and architecture of Storm"?

Case

Count the statistical value of the region where the user ip resides

Typical application scenario

Request reply (synchronization)

Streaming (asynchronous)

-deal with it one by one

There is no relationship between the data: for example, the real-time log format is standardized for storage.

-Analysis and processing

Relationship between data (aggregation, etc.): such as log pv/uv statistics, access hotspot statistics

DGA computing model

Tuple: data processing unit

Stream: continuous tuple flow

Spout: get the data from the outside and output the original tuple

Bolt: receive tuple from spout/bolt output, process and output new tuple

The grouping method of multiple concurrent task from upstream to some downstream of grouping:tuple

-- shuffleGrouping: randomly send it to a downstream task

-- fieldsGrouping: select the hash module according to certain fields and send it to the corresponding task

-- allGrouping: sent to all downstream task

Topology: a spout,bolt,grouping combination of applications

Principle and architecture of Storm

Nimbus: the master of the cluster, which is responsible for managing supervisor and scheduling topology

Supervisor: the worker responsible for running topology

Worker: responsible for actual computing and network communication

Zookeeper: responsible for storing the status of the above modules to make them highly available.

Work is a process and a unit that allocates resources. A worker can have multiple executor,executor threads, which is the unit of processing.

Executor executes spout.nextTuple () or bolt.execute (), calls emit to generate a new tuple, and puts it into the transfer queue of executor

Executor transfer thread put the tuple in his transfer queue into worker transfer queue.

Worker transfer thread sends the tuple in transer queue to the remote worker

Worker receive thread receives data from the network respectively, and puts the tuple to the receive queue of the corresponding executor

Executor receive thread fetches the tuple from its reveive queue and calls bolt.execute ()

Storm actual combat

Start drpc: bin/storm drpc

Start logviewr: bin/storm viewer

JavaAPI

-spout

NextTuple () callback function, loop trigger

Ack (id) callback function, which is triggered when a message is processed successfully

Fail (id) callback function, which is triggered when the message times out

-bolt

Execute (Tuple input) callback function, data trigger

Collector.emit (tuple): sends tuple to the downstream via collector

Collector.ack (tuple): confirm through collector that the input tuple has been processed successfully

You don't have to use java. You have shellBolt.

DAG graphs: acyclic digraphs (Directed Acyclic Graph)

At this point, I believe you have a deeper understanding of "what is the principle and architecture of Storm". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.