Shulou(Shulou.com)06/02 Report--

Work requires access to the jstorm JStorm cluster, which includes two types of nodes: master node (Nimbus) and work node (Suprvisor)

Run Nimbus Daemon on the master node (Nimbus). Nimbus is responsible for receiving Topology submitted by Client, distributing code, assigning tasks to work nodes, monitoring the status of running tasks in the cluster, and so on. The function of Nimbus is similar to JobTracker in Hadoop. Run Supervisor D on the work node (Supervior). Supervior uses subscribe Zookeeper-related data to monitor tasks assigned by Nimbus to start or stop the Worker worker process. Each Worker worker process executes a subset of Topology tasks; tasks of a single Topology are handled cooperatively by Worker worker processes distributed on multiple work nodes.

Https://blog.csdn.net/wwwxxdddx/article/details/49978311

Coordination between Nimbus and Supervisor nodes is achieved through Zookeeper. In addition, both Nimbus and Supervisor are stateless processes themselves, and the state information of Fail Fast;JStorm cluster nodes is either stored in Zookeeper or persisted locally, which means that even if Nimbus/Supervisor is down, it can continue to work after reboot. This design makes the JStorm cluster very stable.

Find out what he does first.

Distributed real-time computing engine, where you give him data, and according to the rules you write, he finishes it.

Standard explanation: the user implements a task according to the specification, and puts the task on jstorm, and the jstorm schedules the task 24 hours a day.

Advantages: development is very fast and easy to use, just like a faucet-"sewage tank treatment -" downstream

Excellent scalability, one worker breaks down, the scheduler nimbus will assign another immediately.

The accuracy is relatively high, and the acker mechanism is adopted to ensure that the data is not lost.

Usage scenarios: log analysis, piping systems, message conversion, statistical analysis

These terms should be known.

Role role

Of the task assigned by the Nimbus scheduler

The agent role of Supervisor Worker, which is responsible for Kill Worker and running Worker sewage tank treatment

Container worker at Worker Task

The executor worker of the Task task

The coordinator of the ZooKeeper system

Stream flow is an uninterrupted continuous tuple meta-value.

The source of the original meta-ancestors of spout, like a faucet, this may be a message middleware such as MQ,Kafka,TBNofity constantly sending messages.

With the source and the tuple, the processing process of holding these data is Bolt.

Bolt consumes any number of input streams, sewage sinks

Your spout is open, if you want to get that type of water, open that type of Bolt, and then proceed to the next step or flow to another container.

Tuple value list values list

A Topology directed acyclic graph is a topology, and a topology is a data flow graph, indicating whether each node is a spout or bolt, and sending meta-ancestor data to that bolt.

Worker and task

Worker represents a process, tash represents a thread, and a process corresponds to multiple threads

Supervisor is the work node in JStorm, which is similar to the task scheduling result data of MR's TT,subscribe zookeeper. It starts / stops the worker process Worker according to the task scheduling situation. At the same time, Supervisor needs to write active port information to zookeeper regularly for Nimbus monitoring. Supervisor does not perform specific processing work, and all computing tasks are handed over to Worker to complete. From the perspective of the whole architecture, Supervisor is in the middle of the three-tier management architecture of JStorm, which assists in the management of task scheduling and resource management.

The Supervisor single-node architecture is shown in the figure above. The process Supervisor is started during initialization, and the Worker JVM process is triggered according to the task assigned by Nimbus. Each Worker process starts one or more task threads, and Task must belong to a single Topology. From the perspective of the entire Supervisor node, multiple JVM processes are running, including a Supervisor process and one or more Worker processes.

Different role states are maintained in different ways. Task writes time information and current Task statistics directly to zookeeper;Worker through hb and periodically writes Topology id, port, Task id collection and current time locally; Supervisor periodically writes time and node resources (port set) to zookeeper, while reading task scheduling results from zookeeper, starting / deactivating Worker processes according to the results.

Remember this picture is basically not difficult to hit, getting started.

To understand this, you have to know the concept of flow, that is, faucets-- "sewage tank treatment--" downstream.

This link is more like https://www.cnblogs.com/smartloli/p/4810362.html.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.