Shulou(Shulou.com)06/01 Report--

This article will explain in detail the example analysis of building a storm environment for a stand-alone version of ubuntu. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

1 reference blog post

Http://www.makaidong.com/%E5%8D%9A%E5%AE%A2%E5%9B%AD%E6%8E%92%E8%A1%8C/5364.shtml

Http://jingpin.jikexueyuan.com/article/36537.html

Http://my.oschina.net/mingdongcheng/blog/43009?fromerr=Mni4Cbek

Http://stackoverflow.com/questions/22216433/java-home-not-set-when-install-storm

Http://www.xuebuyuan.com/1932716.html

2 introduction to Storm

Storm's cluster is very similar to hadoop's cluster. But on hadoop you run mapreduce's job, and on storm you run topology.

There are two types of nodes in the storm cluster: control node (master node) and work node (worker node). A daemon runs on the control node: nimbus, which acts like jobtracker in hadoop. Nimbus is responsible for distributing code in the cluster, assigning work to machines, and monitoring status.

Each work node runs a node called supervisor. Supervisor monitors the work assigned to its machine and starts / shuts down the worker process as needed. Each worker process executes a subset of a topology; a running topology consists of many worker processes running on many machines.

All coordination between nimbus and supervisor is done through a zookeeper cluster.

Stream is the key abstraction in storm. A stream is an unbounded tuple sequence. Storm provides primitives to transfer a stream to a new stream in a distributed and reliable manner. For example, you can stream a tweets to a stream of hot topics.

The most basic primitives for dealing with stream provided by storm are spout and bolt. You can implement the interface between spout and bolt to handle the logic of your application.

The source of the spout stream. For example, a spout might read messages from a kestrel queue and transmit them into a stream.

Bolt can receive any number of input stream for some processing, and some bolt may also emit some new stream. Some complex flow switching, such as figuring out hot topics from some tweet, requires multiple steps, which requires multiple bolt. Bolt can do anything: run functions, filter tuple, do some aggregations, do some merges, access databases, and so on.

A network composed of spout and bolt is packaged into topology. Topology is the highest level of abstraction in storm, which is equivalent to hadoop's mapreduce. You can submit topology to storm's cluster to run.

There is a zeromq component between the entire data flow, which is mainly responsible for messaging, through the

This article comes from: Ma Kaidong blog reproduced, please indicate the source URL: http://www.makaidong.com

Coordination can avoid queuing congestion in data transmission.

Every node in topology runs in parallel. In your topology, you can specify the parallelism of each node, and storm allocates so many threads in the cluster to calculate at the same time.

3 installation preparation

1. Ubuntu 14.04

2. JDK version 1.7

3. Python 2.7

4. Zookeeper 3.4.7

5. ZeroMQ 2.1.7

6. Jzmq

7. Storm 0.10.0

4 installation step 4.1 Linux installation

Because of the limited number of devices, a 64-bit Server version of Ubuntu built by the VMware Workstation virtual machine is used locally. If you have a lot of information on the Internet, you don't have to elaborate on it. It takes some time.

4.2 JDK installation 4.2.1 JDK download

The local selection is under JDK 1.7.The address is: http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

4.2.2 JDK installation

After downloading, put it in the specified location (such as / home/geyg/storm/install) and install it using the following command

Tar zxvf jdk-7u79-linux-x64.tar.gz

For ease of management, you can move the extracted files to another folder.

Sudo mv jdk1.7.0_79/ / usr/lib/jdk1.7/4.2.3 JDK configuration

1. Enter the / etc directory

two。 Add JDK configuration information to the profile file

# The following configuration is for Javaexport JAVA_HOME=/usr/lib/jdk1.7export PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=$JAVA_HOME/lib

3. Use the following combination command to make the configuration effective

Chmod + x profile (this command requires super privileges) source profile

4. Use the following to verify the JDK installation

Java-version

The installation is successful if the installation version information is displayed

Note: JDK configuration can also be configured in ~ / .bashrc. After configuration, source ~ / .bashrc makes it effective.

4.3 Python installation

The Ubuntu system is installed by default, and the version number can be checked by the following shell command:

Python-V

If Python is not already installed on your system, follow these steps to install it:

4.3.1 Python download wget http://www.python.org/ftp/python/2.7.2/python-2.7.2.tgz4.3.2 Python install tar zxvf python-2.7.2.tgzcd python-2.7.2. / configure / / configuration dependency, generate makefilemake / / compile makefilemake install / / install 4.3.3 Python configuration vi / etc/ld.so.conf

At the end of the file / usr/local/lib/

Execute the following command to make Python effective

Sudo ldconfig

At this point, Python is installed.

4.4 zookeeper installation

Objective: to coordinate the whole cluster, mainly to act as the contact intermediary between nimbus and supervisor, and to manage the role of components in the cluster.

4.4.1 zookeeper download wget http://ftp.meisei-u.ac.jp/mirror/apache/dist//zookeeper/zookeeper-3.4.7/zookeeper-3.4.7.tar.gz4.4.2 zookeeper install tar zxvf zookeeper-3.4.7.tar.gzcp-r zookeeper-3.4.7 / usr/local/cd / usr/localmv zookeeper-3.4.7 zookeeper4.4.3 zookeeper configuration vi ~ / .bashrc

The following code is appended to the end of the file

Export ZOOKEEPER_HOME=/usr/local/zookeeperexport PATH=$PATH:$ZOOKEEPER_HOME/bi

Modify the contents of zoo.cfg file

Cp / usr/local/zookeeper/conf/zoo_sample.cfg / usr/local/zookeeper/conf/zoo.cfg

The dataDir in this file is:

DataDir=/usr/local/zookeeper/zookeeperdir/zookeeper-data

Create a directory / usr/local/zookeeper/zookeeperdir/zookeeper-data

Cd / usr/local/zookeepermkdir zookeeperdircd zookeeperdirmkdir zookeeper-data

At this point, the stand-alone installation of zookeeper has been completed.

4.5 ZeroMQ installation

Zeromq is used for underlying communication.

Zeromq is a new layer in the network stack, it is a scalable layer, distributed among distributed systems. Therefore, it can support any large application. Zeromq is not a simple point-to-point interaction; instead, it defines the global topology of a distributed system. Zeromq applications do not have locks and can run in parallel. In addition, it can stretch between multiple threads, kernels, and host boxes.

4.5.1 ZeroMQ download wget http://download.zeromq.org/zeromq-2.1.7.tar.gz4.5.2 ZeroMQ install tar zxvf zeromq-2.1.7.tar.gzcd zeromq-2.1.7./configuremakemake installsudo ldconfig (update ld_library_path)

Note:

Here are some of the problems you might encounter in. / configure:

Configure:error:in'/ usr/local/download/zeromq-2.1.7':

Congifure:error:no acceptable c compiler found in $path

See 'config.log' for more details

This is because the c compiler is not installed.

The solution is: # apt-get install gcc*

The problems encountered after that are: error:cannot link with-luuid, install uuid-dev

This is because the uuid-related package is not installed.

The solution is: # apt-get install uuid*

# apt-get install e2fsprogs *

# apt-get install libuuid*

The problem I encountered is the problem of compiling C++.

Solution.

# apt-get install gaming +

#. / configure

After the problem is solved, you can make and make install, so zeromq is installed, and then you can install jzmq.

4.6 jzmq installation 4.6.1 jzmq download git clone git://github.com/nathanmarz/jzmq.git

/ / if there is no git, you can install it: sudo apt-get install git

4.6.2 jzmq installation. / autogen.sh

/ / A missing package may be reported. Install: sudo apt-get install libtool auto-make first

May report missing: pkg-config tool

Wget http://pkgconfig.freedesktop.org/releases/pkg-config-0.23.tar.gz

Tar zxf pkg-config-0.23.tar.gz

Cd pkg-config-0.23

. / configure-prefix=/usr/local/pkg-config-0.23-datarootdir=/usr/share

Make

Sudo make install

Set PATH: (pkg-config will be used later) after installation is complete.

Export PATH=.:/usr/local/pkg-config-0.23/bin:$PATH

(note: please choose the installation of pkg-config-0.23.tar.gz or previous version. I chose the latest version of 0.25,0.26. Make has failed for a long time. Please don't fall into this pit again. The installation of the previous version of pkg-config-0.23 is also a bit of a trap, please see below)

Small Trap-install pkg-config

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.