Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to build a Twitter Storm system cluster. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What is Storm?

Storm is a distributed real-time computing system open source by Twitter.

Use the scene:

Real-time data analysis, continuous computing, distributed RPC and so on.

Storm features (Storm is similar to a walking elevator, it will run all the time without failure, while hadoop is similar to an elevator and will stop to a certain extent. ):

Distributed system

Scalable

High reliability

The programming model is simple

Efficient real-time

Commonly used classes:

BaseRichSpout (message producer)

BaseBasicBolt (message handler)

TopologyBuilder (the builder of the topology)

Config (configuration)

StormSubmitter/LocalCluster (Topology Breaker)

Storm cluster deployment

Storm cluster architecture:

As shown in the figure:

There are two types of nodes in the Storm cluster: control nodes and work nodes.

A process called Nimbus runs on the control node, and Nimbus is responsible for running in the cluster.

Distribute code, assign computing tasks, and monitor status.

Each worker node runs a process called Supervisor.

The Supervisor is responsible for listening on the tasks assigned to it from the Nimbus, thereby starting or stopping the worker process that executes the task.

All coordination between Nimbus and Supervisor is done through the Zookeeper cluster.

Cluster planning: (planning according to specific needs)

Linux hostname Storm role Zookeepermaster Nimubus single node zkslave01 Supervisorslave02 Supervisor

Preparatory work:

Environment: centos6.4

Software:

Jzmq-masterstorm-0.8.2zeromq-2.1.7zookeeper-3.4.5

Environment configuration: (see previous blogs)

Basic Linux configuration:

Modify hostname

Modify IP

Modify the mapping relationship between host and IP

Turn off the firewall

Installation steps:

1. Install jdk

two。 Set up a Zookeeper cluster (here we only install a zk on the primary node)

Decompression

Enter the conf directory of zk and cp zoo_sample.cfg zoo.cfg (change the name). Everything else remains the same for the time being.

3. Install Storm dependencies (zeromq, jzmq, python)

3.1 install zeromq, then go to the zeromq-2.1.7/ directory

Detection environment:. / configurecd zeromq-2.1.7./configure

# there may be errors in compilation:

Configure: error: Unable to find a working C++ compiler

# install the dependent rpm package:

Libstdc++-devel gcc-c++

When the virtual machine can access the Internet: (this method is recommended)

Yum install gcc-c++

The virtual machine cannot access the Internet:

First go to http://mirrors.163.com/centos/6.4/os/x86_64/Packages/ (the downloaded version must correspond to the system)

Rpm-I libstdc++-devel-4.4.7-3.el6.x86_64.rpmrpm-I gcc-c++-4.4.7-3.el6.x86_64.rpmrpm-I libuuid-devel-2.17.2-12.9.el6.x86_64.rpm

Then run. / configure

Make (compiled)

Make install (this is only then thoroughly installed)

3.2. Compile and install JZMQ:

Cd jzmq

Execute. / autogen.sh (in order for it to generate a configuration file, which is not available by default)

# error report: autogen.sh: error: could not find libtool.libtool is required to run autogen.sh. Missing libtool

Similarly, in the case of Internet access:

Yum install libtool (readhat Enterprise Edition does not have these errors)

Or manually install:

Rpm-I autoconf-2.63-5.1.el6.noarch.rpmrpm-I automake-1.11.1-4.el6.noarch.rpmrpm-I libtool-2.2.6-15.5.el6.x86_64.rpm./configuremakemake install

3.33. Compile and install Python (first determine the version that comes with your system, if it is 2.6.6 or above, it does not need to be installed)

Tar-zxvf Python-2.6.6.tgzcd Python-2.6.6./configuremakemake install

3.4 install storm

Modify the storm.yaml configuration file (also have to be modified on the child node) modify the hostname corresponding to zk modify the hostname corresponding to the primary node

PS:

3.41 under the unzipped directory of the Storm distribution, there is a

Conf/storm.yaml file:

Used to configure Storm. Default configuration can be seen here that the configuration options in conf/storm.yaml will override the default configuration in defaults.yaml.

The following configuration options must be configured in conf/storm.yaml:

Storm.zookeeper.servers:

The Zookeeper cluster address used by the Storm cluster

The format is as follows:

Storm.zookeeper.servers:- "111.222.333.444"-"555.666.777.888"

If the Zookeeper cluster does not use the default port, then the storm.zookeeper.port option is also required.

3.42 storm.local.dir: Nimbus and Supervisor processes

The local disk directory used to store a small amount of state, such as jars, confs, etc., needs to be created in advance and given sufficient access rights.

Then configure the directory in storm.yaml, such as:

Storm.local.dir: "/ usr/storm/workdir"

Start each of the three machines, master: to the bin directory of storm:

. / storm nimbus > / dev/null 2 > & 1 &

Slave01: go to the bin directory of storm:

. / storm supervisor >.. / logs/su.log 2 > & 1 &

Slave02: go to the bin directory of storm:

. / storm supervisor >.. / logs/su.log 2 > & 1 &

(start the background process and output correct and incorrect information to the file)

Launch the UI management interface on master

. / storm ui > / dev/null 2 > & 1 &

Observe through the browser: (ip:8080 of the master node), observe the worker resource usage of the cluster, the running status of Topologies and other information.

The Storm cluster has been deployed and configured, and you can submit the topology to the cluster to run.

The above is the editor for you to share how to build a Twitter Storm system cluster, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.