Shulou(Shulou.com)06/01 Report--

This article mainly explains "the method of building and configuring Storm cluster". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the method of building and configuring Storm cluster".

Install Storm cluster

This section describes in detail how to build a Storm cluster. Here are the installation steps that need to be completed in turn:

Set up Zookeeper cluster

Install Storm dependent libraries

Download and extract the Storm release

Modify storm.yaml configuration file

Start each background process of Storm.

1. Reference for building Zookeeper: http://my.oschina.net/u/2377453/blog/464739

Considerations for building a Zookeeper cluster:

Because Zookeeper fails quickly (fail-fast), and the process exits when any error occurs, it is best to manage the Zookeeper through the monitor program to ensure that the Zookeeper can be automatically restarted after exiting. Please refer to here for details.

During the running of Zookeeper, a lot of log and snapshot files are generated in the dataDir directory, and the Zookeeper running process is not responsible for cleaning and merging these files on a regular basis, which takes up a lot of disk space. Therefore, it is necessary to clean up useless log and snapshot files regularly through cron and other methods. Please refer to here for details. The specific command format is as follows: java-cp zookeeper.jar:log4j.jar:conf org.apache.zookeeper.server.PurgeTxnLog-n

2. Install the Storm dependent library

Next, you need to install Storm's dependent libraries on Nimbus and Supervisor machines, as follows:

ZeroMQ 2.1.7-do not use version 2.1.10, as some serious bug in this version can cause strange problems when the Storm cluster is running. A small number of users will encounter a "IllegalArgumentException" exception in version 2.1.7, which can be fixed by reducing it to version 2.1.4.

JZMQ

Java 6

Python 2.6.6

Unzip

The versions of the above dependent libraries have been tested by Storm, and Storm is not guaranteed to run under other versions of Java or Python libraries.

Note:

Dependencies often occur during installation, be sure to use yum to install various dependencies

If there is a problem with YUM, please refer to this article: http://my.oschina.net/u/2377453/blog/466141

3. Install ZMQ 2.1.7

Download and compile and install ZMQ:

# wget / / http://download.zeromq.org/zeromq-2.1.7.tar.gz// download # tar-xzf zeromq-2.1.7.tar.gz / / extract # cd zeromq-2.1.7 / / change directory / / compile and install #. / configure # make # make install installation may occur the following dependencies: 1. Install C++ compiler # sudo yum install gcc-c++2. Download libsodium// https://github.com/jedisct1/libsodium/archive/1.0.1.tar.gz3. Install libtool#sudo yum install libtool4. Install autoconf-2.68#wget / / http://ftp.gnu.org/gnu/autoconf/autoconf-2.68.tar.gz#./configure#make & & make install

Note:

1. If the installation process reports that uuid cannot be found, install the uid library with the following package: # yum install uuid*#yum install e2fsprogs*2. If the C++ compilation environment # yum install gcc-c++ appears

4. Install JZMQ

Download and compile and install JZMQ:

# wget / / https://codeload.github.com/zeromq/jzmq/zip/master// download # unzip jzmq-master.zip / / decompress # cd jzmq-maste / / change directory / / compile and install #. / autogen.sh#./configure#make#make install

In order for JZMQ to work properly, you may need to complete the following configuration:

Set the JAVA_HOME environment variable correctly

Install the Java package

Note:

1. If connot find zmq.h appears when you run the. / configure command

Join two export:export CPPFLAGS=-I/root/installs/zeromq/includeexport LDFLAGS=-L/root/installs/zeromq/lib/

5. Install Java 6

1. Download and install JDK 6

two。 Configure JAVA_HOME environment variables

3. Run the java, javac commands to test the normal installation of java.

6. Install Python2.7.10

1. Download Python2.7.10:

Wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tar.xz

two。 Compile and install Python2.7.10:

# xz-d Python-2.7.10.tar.xz// may need to install xz Sudo yum install xz # tar-xvf Python-2.7.10.tar.gz # cd Python-2.7.10#./configure#make#make install Note: 1. The default installation path of python is / usr/local/bin/pythonN2. To replace the python that comes with the system, the newly installed python must delete the link file of the python that comes with the system: / usr/bin/python3, Use the command to create your own link file: ln-s / usr/local/bin/pythonN / usr/bin/pythonN bit version number

3. Test Python2.7.10:

# python-VPython 2.7.10

7. Install unzip

1. If you are using a RedHat series Linux system, execute the following command to install unzip:

Apt-get install unzip

two。 If you are using a Debian series Linux system, execute the following command to install unzip:

Yum install unzip

8. Download and extract the Storm release

Next, you need to install the Storm distribution on Nimbus and Supervisor machines.

1. To download the Storm distribution, it is recommended to use Storm0.8.1:

Wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip

two。 Unzip it to the installation directory:

Unzip storm-0.8.1.zip

3. Modify storm.yaml configuration file

Under the unzipped directory of the Storm distribution, there is a conf/storm.yaml file that is used to configure Storm. The default configuration can be seen here. The configuration options in conf/storm.yaml override the default configuration in defaults.yaml. The following configuration options must be configured in conf/storm.yaml:

Note: there must be spaces on the left and right sides of ":" and "-" in the configuration file (→ _ → I was cheated for a long time to find this problem)

# configure the address of the zookeeper cluster used by storm storm.zookeeper.servers:-"hostA"-"hostB" # (optional) if the default port of zookeeper is not 2181, you need to configure zookeeper port storm.zookeeper.port: 218 configure the host where the primary node nimbus is located nimbus.host: "host" # configure storm's working directory storm.local.dir: "/ root/app/storm/tmp" # configure whether ZeroQM is used as the messaging system in local mode Default is false: use java message system storm.local.mode.zmq: true# to configure the supervisor node can use the port of worker process. The default is 4 worker processes. The port is as follows:-6700-6701-6702-6703.

9. Start each background process of Storm

The last step is to start all background processes for Storm. Like Zookeeper, Storm is a fail-fast system so that Storm can be stopped at any time and resume execution correctly when the process is restarted. This is why Storm does not save state in the process, and even if Nimbus or Supervisors is restarted, the running Topologies will not be affected.

Here is how to start each background process in Storm:

Nimbus: run "bin/storm nimbus 1 > / dev/null 2 & 1 &" on the Storm master node to start the Nimbus daemon and put it into the background to execute

Supervisor: run "bin/storm supervisor 1 / dev/null 2 & 1 &" on each worker node of Storm to start the Supervisor daemon and put it into the background to execute

UI: run "bin/storm ui 1 > / dev/null 2 & 1 &" on the Storm master node to start the UI daemon and put it into the background to execute. After startup, you can observe the cluster worker resource usage, Topologies running status and other information through http://{nimbus host}: 8080.

Command interpretation: "1 > / dev/null 2 & 1", linux command, meaning standard output (1) to / dev/null, exception output (2) is also output to / dev/null (& 1 is the location of reference 1 output), & running in the background, if you add "nohub" to the front end of the command, it will continue to run even if the terminal is closed.

Note:

After the Storm background process is started, log files for each process are generated in the logs/ subdirectory under the Storm installation and deployment directory.

It has been tested that Storm UI must be deployed on the same machine as Storm Nimbus, otherwise UI will not work properly because the UI process will check for native Nimbus links.

For ease of use, bin/storm can be added to the system environment variable.

At this point, the Storm cluster has been deployed and configured, and the topology can be submitted to the cluster for operation.

10. Submit tasks to the cluster

1) start Storm Topology:

Storm jar allmycode.jar org.me.MyTopology arg1 arg2 arg3

Where allmycode.jar is the jar package containing the Topology implementation code, the main method of org.me.MyTopology is the entry of Topology, and arg1, arg2 and arg3 are the parameters that need to be passed in when org.me.MyTopology executes.

2) stop Storm Topology:

Storm kill {toponame}

Where {toponame} is the name of the Topology task specified when Topology is submitted to the Storm cluster.

Thank you for your reading. the above is the content of "how to build and configure Storm clusters". After the study of this article, I believe you have a deeper understanding of the method of building and configuring Storm clusters, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.