Shulou(Shulou.com)06/01 Report--

This article mainly explains "Storm's operation and directory structure on zookeeper". The explanation content in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian to study and learn "Storm's operation and directory structure on zookeeper" together.

Zookeeper operations

(defprotocol ClusterState (set-ephemeral-node [this path data]) (delete-node [this path]) (create-sequential [this path data]) (set-data [this path data]) ;; if node does not exist, create persistent with this data (get-data [this path watch?]) (get-children [this path watch?]) (mkdirs [this path]) (close [this]) (register [this callback]) (unregister [this id]) )

Storm uses Zookeeper operations

(defprotocol StormClusterState (assignments [this callback]) (assignment-info [this storm-id callback]) (active-storms [this]) (storm-base [this storm-id callback]) (get-worker-heartbeat [this storm-id node port]) (executor-beats [this storm-id executor->node+port]) (supervisors [this callback]) (supervisor-info [this supervisor-id]) ;; returns nil if doesn't exist (setup-heartbeats! [this storm-id]) (teardown-heartbeats! [this storm-id]) (teardown-topology-errors! [this storm-id]) (heartbeat-storms [this]) (error-topologies [this]) (worker-heartbeat! [this storm-id node port info]) (remove-worker-heartbeat! [this storm-id node port]) (supervisor-heartbeat! [this supervisor-id info]) (activate-storm! [this storm-id storm-base]) (update-storm! [this storm-id new-elems]) (remove-storm-base! [this storm-id]) (set-assignment! [this storm-id info]) (remove-storm! [this storm-id]) (report-error [this storm-id task-id error]) (errors [this storm-id task-id]) (disconnect [this]) )

Directory stored in Zookeeper in Storm

(def ASSIGNMENTS-ROOT "assignments")(def CODE-ROOT "code")(def STORMS-ROOT "storms")(def SUPERVISORS-ROOT "supervisors")(def WORKERBEATS-ROOT "workerbeats")(def ERRORS-ROOT "errors")(def ASSIGNMENTS-SUBTREE (str "/" ASSIGNMENTS-ROOT))(def STORMS-SUBTREE (str "/" STORMS-ROOT))(def SUPERVISORS-SUBTREE (str "/" SUPERVISORS-ROOT))(def WORKERBEATS-SUBTREE (str "/" WORKERBEATS-ROOT))(def ERRORS-SUBTREE (str "/" ERRORS-ROOT))

/assignments -> task assignment information

/storms -> ID of running topology

/supervisors -> heartbeat information for all Supervisors

/workerbeats -> Heartbeats of all Workers

/errors -> error messages generated

structure diagram

/-{storm-zk-root} -- storm Root directory on zookeeper (default is/storm)||-/assignments --task assignment information for topology| | | |-/{topology-id} --This directory stores the assignments information of each topology, including: corresponding nimbus| --the code directory, the startup time of all tasks, and the mapping of each task to machines and ports. operable to| -- (assignments) to get all assignments values; and (assignment-info storm-id)| --to obtain Assignment Info information corresponding to the given storm-id| --The contents stored in Assignment Info are:| -- :executor->node+port :executor->start-time-secs :node->host | --specifically defined in common.clj.| -- (defrecord Assignment[master-code-dir node->host executor->node+port executor->start-time-secs]) | |-/storms --This directory holds the ids of all running topologies| | | | | |-/{topology-id} --This file holds some information about this topology, including the name of the topology, and the topology starts running.| time and the state of the topology. Operation (active-storms), get the current path active under| topology data. The contents saved refer to the class StormBase;(storm-base storm-id) to get the given| StormBase data under--storm-id, specifically defined in common.clj| -- (defrecord StormBase [storm-name launch-time-secs status num-workers component->executors]) | |-/supervisors --This directory holds all supervisor heartbeat information| | | | | |-/{supervisor-id} --This file holds the supervisor's heartbeat information including: heartbeat time, host name, on this supervisor| --Port number of worker, runtime (see Supervisor Info class for details). operation (supervisors) to obtain| --all supervisors nodes;(supervisor-info supervisor-id) get the given| --SupervisorInfo information corresponding to supervisor-id; specifically defined in common.clj| | -- (defrecord SupervisorInfo [time-secs hostname assignment-id used-ports meta scheduler-meta uptime-secs]) | |-/workerbeats --Heartbeat of all workers| | | |-/{topology-id} --This directory holds heartbeat information for all workers of this topology| | | |-/{supervisorId-port} --Heartbeat information of worker, including heartbeat time, worker runtime and some statistics| | --operation (heartbeat-storms) to get all the topology with heartbeat data,| -- (get-worker-heartbeat storm-id node port) Get a specific topology under| --The heartbeat status of a worker(node:port) of| -- (executor-beats storm-id executor->node+port) Get an executor heartbeat status||-/errors Thank you for reading, the above is the content of "Storm operation and directory structure on zookeeper", after learning this article, I believe everyone has a deeper understanding of Storm operation and directory structure on zookeeper, and the specific use needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.