Shulou(Shulou.com)06/01 Report--

This article mainly explains "the way of building Storm cluster". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the way of building Storm cluster".

Build Storm cluster environment Step 1. 0 under Ubuntu 14.04. Download the required software packages:

1)。 JDK: http://www.oracle.com/technetwork/java/javase/downloads/index.html

2)。 Python 2.7.x: https://www.python.org/downloads/

(note: because Ubuntu 14.04 comes with Python 2.7, this step can be avoided if the Storm cluster environment is built under Ubuntu 14.04.)

3)。 Zookeeper: http://zookeeper.apache.org/releases.html#downloa

4)。 Storm: http://storm.apache.org/downloads.html

Step 2. Install JDK

Assume that the downloaded JDK file is jdk-8u65-linux-x64.tar.gz

Copy the file to / usr/local/ and extract:

$cp jdk-8u65-linux-x64.tar.gz / usr/local/$ cd / usr/local/$ tar-zxvf jdk-8u65-linux-x64.tar.gz$ ln-s jdk1.8.0_65 jdk

Edit the ~ / .bashrc file and set the environment variables:

$vim ~ / .bashrc

(* Note: if you have not previously installed vim, you need to enter the following command to install vim)

$sudo apt-get install vim

Add the environment variable setting at the end of the ~ / .bashrc file:

Export JAVA_HOME=/usr/local/jdkexport JRE_HOME=$ {JAVA_HOME} / jreexport CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / libexport PATH=$PATH:$ {JAVA_HOME} / bin:

Update the settings of the environment variable:

$source ~ / .bashrc

Enter the command in the terminal to display the information of JDK:

$java-version

If you output the following information at this time, JDK has been installed successfully

Java version "1.8.065" Java (TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot (TM) 64-Bit Server VM (build 25.65-b01, mixed mode) Step 3. Configure Zookeeper1) configuration / etc/hosts file

First configure the / etc/hosts file to bind each machine in the cluster (assuming that only two machines are used here):

$vim / etc/hosts

Add configuration information at the end of the / etc/hosts file:

192.168.1.104 jiamingmai-notebook192.168.1.104 jm01192.168.1.105 jiamingmai-PHA14192.168.1.105 jm02

(* Note: set according to your hostname and IP address, where jiamingmai-notebook and jiamingmai-PHA14 are hostnames, 192.168.1.104 and 192.168.1.105 are IP addresses of these two machines, and jm01 and jm02 are custom names. If you don't know the IP address of the machine, you can check it using the ifconfig command)

After configuring the / etc/hosts file, you can use the ping command on each of the two machines to see if it is connected:

$ping jm01 $ping jm02 $ping jiamingmai-notebook$ ping jiamingmai-PHA14

If you output a response message similar to the following, it means that you can successfully connect:

64 bytes from jiamingmai-PHA14 (192.168.1.105): icmp_seq=1 ttl=64 time=2149 ms64 bytes from jiamingmai-PHA14 (192.168.1.105): icmp_seq=2 ttl=64 time=1142 ms64 bytes from jiamingmai-PHA14 (192.168.1.105): icmp_seq=3 ttl=64 time=134 ms64 bytes from jiamingmai-PHA14 (192.168.1.105): icmp_seq=4 ttl=64 time=2.06 ms64 bytes from jiamingmai-PHA14 (192.168.1.105): icmp_seq=5 ttl=64 time=2.07 ms2) configure conf for Zookeeper

Copy the compressed package zookeeper-3.4.6.tar.gz to / usr/local, and extract:

$cp zookeeper-3.4.6.tar.gz / usr/local/$ cd / usr/local$ tar-zxvf zookeeper-3.4.6.tar.gz$ ln-s zookeeper-3.4.6 zookeeper

Configure zookeeper:

$cd zookeeper/conf$ cp zoo_sample.cfg zoo.cfg$ vim zoo.cfg

Configure the zoo.cfg text as follows:

TickTime=2000initLimit=10syncLimit=5dataDir=/usr/local/zookeeper/zkdatadataLogDir=/usr/local/zookeeper/logsclientPort=2181server.1=jm01:2888:3888server.2=jm02:2888:3888

For a specific description of each parameter, please see http://my.oschina.net/JiamingMai/blog/614246

Create the folder specified by dataDir and dataLogDir:

$mkdir / usr/local/zookeeper/zkdata$ mkdir / usr/local/zookeeper/logs

Create a myid file under the folder specified by dataDir (/ usr/local/zookeeper/zkdata in this case) with the contents of the id corresponding to the machine. For example, for jm01, the id is 1. 0. Enter the following command to create the myid file:

$cd / usr/local/zookeeper/zkdata$ echo 1 > myid3) sets the environment variable of Zookeeper

Enter the following command to edit the ~ / .bashrc file:

$vim ~ / .bashrc

Add the following at the end:

Export ZOOKEEPER_HOME=/usr/local/zookeeperexport PATH=$PATH:$ {ZOOKEEPER_HOME} / bin

Enter the following command to update the environment variable settings:

$source ~ / .bashrc4) run Zookeeper

Enter the following command on the machines in the cluster to run Zookeeper:

$zkServer.sh start

View the current status on each machine:

$zkServer.sh status

If you see the output Mode: follower or Mode: leader, it indicates that it has been started successfully

Step 4. Configure Storm1) extract the compressed files of Storm

Copy the compressed package apache-storm-0.10.0.tar.gz to / usr/local, and extract:

$cp apache-storm-0.10.0.tar.gz / usr/local/$ cd / usr/local$ tar-zxvf apache-storm-0.10.0.tar.gz$ ln-s apache-storm-0.10.0 storm2) configure conf$ vim / usr/local/storm/conf/storm.yaml of Storm

Enter the content:

Storm.zookeeper.servers:-jm01-"jm02" nimbus.host: "jm01" storm.local.dir: "/ usr/local/storm/status" supervisor.slots.ports:-6700-6701-6702-6703

(* Note: for the meaning of each parameter, please see http://my.oschina.net/JiamingMai/blog/536877.. Storm.zookeeper.servers settings and nimbus settings must be the same on each machine)

Create the folder specified by storm.local.dir:

$mkdir / usr/local/storm/status3) set the environment variable of Storm

Enter the following command to edit the ~ / .bashrc file:

$vim ~ / .bashrc

Add the following at the end:

Export STORM_HOME=/usr/local/zookeeperexport PATH=$PATH:$ {STORM_HOME} / bin

Enter the following command to update the environment variable settings:

$source ~ / .bashrc4) run Storm

Run on the machine as nimbus:

$storm nimbus > / dev/null 2 > & 1 &

Run on the machine as supervisor:

$storm supervisor > / dev/null 2 > & 1 &

(* Note: as a nimbus machine, it can also be used as a supervisor)

Turn on Storm UI on one of the machines:

$storm ui > / dev/null 2 > & 1 &

Enter 127.0.0.1 URL 8080 in the browser to open the UI page, as shown in the following figure:

Step 5. Submit JAR files to the cluster

Use the following command to submit the packaged JAR package to the cluster:

$storm jar. / WordCounterTopology.jar jm.app.storm.WordCounterTopology wordCounter

Where. / WordCounterTopology.jar is the path where the JAR file is located, jm.app.storm.WordCounterTopology is the class name of the main method, and wordCounter is the Topology name that was customized at the time of submission

Thank you for your reading. the above is the content of "how to build Storm clusters". After the study of this article, I believe you have a deeper understanding of how to build Storm clusters, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.