Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build a Storm cluster for you. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

What is Storm?

Storm is a distributed real-time computing system open source by Twitter.

Use the scene:

Real-time data analysis, continuous computing, distributed RPC and so on.

Storm features: (Storm is similar to a walking elevator, it will run all the time without failure.

Hadoop is like an elevator, it will stop when it reaches a certain degree)

Distributed system

Scalable

High reliability

The programming model is simple

Efficient real-time

Commonly used classes:

BaseRichSpout (message producer)

BaseBasicBolt (message handler)

TopologyBuilder (the builder of the topology)

Config (configuration)

StormSubmitter/LocalCluster (Topology Breaker)

Storm cluster deployment

Storm cluster architecture:

As shown in the figure:

There are two types of nodes in the Storm cluster: control nodes and work nodes.

A process called Nimbus runs on the control node, and Nimbus is responsible for running in the cluster.

Distribute code, assign computing tasks, and monitor status.

Each worker node runs a process called Supervisor.

Supervisor is responsible for listening on tasks assigned to it from Nimbus.

Accordingly, start or stop the working process of executing the task.

All coordination between Nimbus and Supervisor is done through the Zookeeper cluster.

Cluster planning: (planning according to specific needs)

Linux hostname Storm role Zookeeper

Master Nimubus single node zk

Slave01 Supervisor

Slave02 Supervisor

Preparatory work:

Environment: centos6.4

Software:

Jzmq-master

Storm-0.8.2

Zeromq-2.1.7

Zookeeper-3.4.5

Environment configuration: (see previous blogs)

Basic linux configuration:

Modify hostname

Modify IP

Modify the mapping relationship between host and IP

Turn off the firewall

Installation steps:

1. Install jdk

two。 Set up a Zookeeper cluster (here we only install a zk on the primary node)

Decompression

Go to the conf directory of zk and cp zoo_sample.cfg zoo.cfg (change the name)

Everything else remains the same for the time being.

3. Install Storm dependencies (zeromq, jzmq, python)

3.1 install zeromq, then go to the zeromq-2.1.7/ directory

Detection environment:. / configure

Cd zeromq-2.1.7

. / configure

# there may be errors in compilation:

Configure: error: Unable to find a working C++ compiler

# install the dependent rpm package: libstdc++-devel gcc-c++

When the virtual machine can access the Internet: (this method is recommended)

Yum install gcc-c++

The virtual machine cannot access the Internet:

First of all to

Http://mirrors.163.com/centos/6.4/os/x86_64/Packages/

(the downloaded version must correspond to the system)

Rpm-I libstdc++-devel-4.4.7-3.el6.x86_64.rpm

Rpm-I gcc-c++-4.4.7-3.el6.x86_64.rpm

Rpm-I libuuid-devel-2.17.2-12.9.el6.x86_64.rpm

Then run. / configure

Make (compiled)

Make install (this is only then thoroughly installed)

3.2. Compile and install JZMQ:

Cd jzmq

Execute. / autogen.sh

(in order for it to generate a configuration file, which is not available by default)

# error report: autogen.sh: error: could not find libtool.

Libtool is required to run autogen.sh.

Missing libtool

Similarly, if you can access the Internet,

Yum install libtool (readhat Enterprise Edition does not have these errors)

Or install it manually

Rpm-I autoconf-2.63-5.1.el6.noarch.rpm

Rpm-I automake-1.11.1-4.el6.noarch.rpm

Rpm-I libtool-2.2.6-15.5.el6.x86_64.rpm

. / configure

Make

Make install

3.33. Compile and install Python

(first determine the version that comes with your system, if it is 2.6.6 or above, it does not need to be installed.)

Tar-zxvf Python-2.6.6.tgz

Cd Python-2.6.6

. / configure

Make

Make install

3.4 install storm

Modify the storm.yaml configuration file (also on child nodes)

Modify the hostname corresponding to zk

Modify the hostname corresponding to the primary node

PS:

Under the unzipped directory of the 3.41.Storm distribution, there is a

Conf/storm.yaml file:

Used to configure Storm. The default configuration can be viewed here

In conf/storm.yaml

Configuration options override the default configuration in defaults.yaml.

The following configuration options are required in the

Configured in conf/storm.yaml:

Storm.zookeeper.servers:

The Zookeeper cluster address used by the Storm cluster

The format is as follows:

Storm.zookeeper.servers:

-"111.222.333.444" >-"555.666.777.888"

If the Zookeeper cluster is not using the default port

Then you also need the storm.zookeeper.port option.

3.42storm.local.dir: Nimbus and Supervisor processes

Used to store a small number of statu

Local disk directories such as jars, confs, etc.

The directory needs to be created in advance and given sufficient access.

Then configure the directory in storm.yaml, such as:

Storm.local.dir: "/ usr/storm/workdir"

Start three machines separately

Master: go to the bin directory of storm

. / storm nimbus > / dev/null 2 > & 1 &

Slave01: go to the bin directory of storm

. / storm supervisor >.. / logs/su.log 2 > & 1 &

Slave02: go to the bin directory of storm

. / storm supervisor >.. / logs/su.log 2 > & 1 &

(start the background process and output correct and incorrect information to the file)

Launch the UI management interface on master

. / storm ui > / dev/null 2 > & 1 &

Observe through the browser: (ip:8080 of the primary node)

Observe the worker resource usage of the cluster,

Information such as the running status of Topologies.

This is the end of the article on "how to build Storm clusters". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.