Shulou(Shulou.com)06/03 Report--

What is Storm:

If only one sentence is used to describe storm, it might be like this: a distributed real-time computing system. According to the authors of storm, storm is similar to what hadoop is to batch processing for real-time computing. As we all know, hadoop implemented according to google mapreduce provides us with the map, reduce primitive, which makes our batch program very simple and elegant. Compared with Hadoop batch processing, Storm is a real-time, distributed and highly fault-tolerant computing system. Like Hadoop, Storm can also handle large amounts of data, but Storm can also make processing more real-time while ensuring high reliability; that is, all information will be processed. Storm can be extended to different machines for mass data processing, and it also has some other features.

The architecture of Storm:

The cluster of Storm consists of a master node and multiple working nodes. The primary node runs a daemon called "nimbus" for assigning code, scheduling tasks, and fault detection. Each worker node runs a daemon called "supervisor" to listen for work, start and terminate the worker process. Both Nimbus and Supervisor can fail quickly and are stateless, so they both become very robust, and the coordination between the two is done by Zookeeper. Zookeeper is used to manage the different components in the cluster. ZeroMQ is the internal messaging system, and JZMQ is the Java Binding of ZeroMQ. There is a subproject called storm-deploy, where you can deploy a Storm cluster with one click on AWS.

Storm advantages:

a. A simple programming model. Similar to MapReduce, which reduces the complexity of parallel batch processing, Storm reduces the complexity of real-time processing.

b. Service-oriented, a service framework that supports hot deployment, instant online or offline App.

c. A variety of programming languages are available. You can use various programming languages on top of Storm. Clojure, Java, Ruby, and Python are supported by default. To increase support for other languages, you only need to implement a simple Storm communication protocol.

d. Fault tolerance. Storm manages worker processes and node failures.

e. Scale horizontally. Computing is performed in parallel among multiple threads, processes, and servers.

f. Reliable message processing. Storm ensures that each message is fully processed at least once. When a task fails, it retries the message from the message source.

g. fast. The design of the system ensures that messages can be processed quickly, and ZeroMQ is used as its underlying message queue.

h. Local mode. Storm has a "local mode" that completely simulates the Storm cluster during processing. This allows you to do development and unit testing quickly.

Problems with Storm:

A. In the current open source version, there is only a single-node nimbus, and hanging up can only restart automatically. You can consider implementing a dual-nimbus layout.

B, Clojure is a dynamic functional programming language running on the JVM platform, the advantage lies in process calculation, part of the core content of Storm is written by Clojure, although the performance improves a lot, but also increases the maintenance cost.

Applicable scenarios for storm:

Stream data processing. Storm can be used to process a steady stream of messages and then write the results to some storage.

Distributed rpc. Because the processing components of storm are distributed and the processing latency is very low, it can be used as a general distributed rpc framework. Of course, our search engine itself is also a distributed rpc system.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.