Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "installation and deployment of Storm On Yarn". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Install JDK7 and Maven

two。 Deploy the Hadoop2 cluster and start yarn

Http://my.oschina.net/zc741520/blog/362824

3. Download Storm on Yarn

[grid@hadoop4 ~] $wget https://github.com/yahoo/storm-yarn/archive/master.zip

4. Compile

[grid@hadoop4 ~] $unzip master.zip [grid@hadoop4 ~] $cd storm-yarn-master## modify pom.xml and change the version number of Hadoop to the corresponding version number [grid@hadoop4 storm-yarn-master] $vim pom.xml 0.9.0-wip21 2.5.2 # # compile [grid@hadoop4 storm-yarn-master] $mvn package-DskipTests

5. Decompress storm-yarn-master/lib/storm-0.9.0-wip21.zip to the upper directory storm-yarn-master.

[grid@hadoop4 storm-yarn-master] $cd lib [grid@hadoop4 lib] $unzip storm-0.9.0-wip21.zip-d. [grid@hadoop4 storm-yarn-master] $lsbin CLA.pdf create-tarball.sh lib LICENSE.txt pom.xml README.md src storm-0.9.0-wip21 target

6. Create the corresponding Storm directory in HDFS

[grid@hadoop4 storm-yarn-master] $hadoop fs-mkdir-p / lib/storm/0.9.0-wip21

7. Put storm.zip on HDFS

# # according to the actual needs, add additional Jar packages required by the Storm project to the lib of storm-0.9.0-wip21, recompress them into storm.zip files, and upload them to the specified directory of HDFS (very important, access the storm.zip in hdfs to obtain the working environment in the cluster)

[grid@hadoop4 storm-yarn-master] $hadoop fs-put. / lib/storm.zip / lib/storm/0.9.0-wip21 [grid@hadoop4 storm-yarn-master] $hadoop fs-ls / lib/storm/0.9.0-wip21Found 1 items-rw-r--r-- 2 grid supergroup 17141078 2015-05-24 19:43 / lib/storm/0.9.0-wip21/storm.zip

8. Some environment variables for hadoop have been set when Hadoop is installed, so now add the following environment variables

[grid@hadoop4 storm-yarn-master] $vim ~ / .bash_profile export PATH=$PATH:/home/grid/storm-yarn-master/storm-0.9.0-wip21/bin:/home/grid/storm-yarn-master/bin [grid@hadoop4 storm-yarn-master] $source ~ / .bash_profile

9. Modify the storm-yarn-master/storm-0.9.0-wip21/conf/storm.yaml configuration file to increase the configuration of zookeeper

# # installation and startup of zookeeper cluster has been described in detail when introducing Storm cluster installation (http://my.oschina.net/zc741520/blog/399233), so I won't repeat it here

[grid@hadoop4 storm-yarn-master] $vim storm-0.9.0-wip21/conf/storm.yaml storm.zookeeper.servers:-"hadoop4"-"hadoop5"-"hadoop6" master.initial-num-supervisors: 1master.container.size-mb: 1024

10. Start the storm on yarn environment

[grid@hadoop4 storm-yarn-master] $storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml

Because storm runs on the cluster as a yarn program, there will be an AppId in the cluster management page of YARN

PS: failed to start the first time, but finally found that it was caused by insufficient memory. The solution is to set the value of yarn.nodemanager.vmem-check-enabled to false in yarn-site.xml.

11. Find the nimbus node

[grid@hadoop4 storm-yarn-master] $storm-yarn getStormConfig-appId application_1432484548277_0001-output ~ / .storm / storm.yaml [grid@hadoop4 storm-yarn-master] $cat ~ / .storm/storm.yaml | grep nimbus.hostnimbus.host: 192.168.0.107

twelve。 Submit Topology, which uses the WordCount example program that comes with Storm.

[grid@hadoop4 storm-yarn-master] $storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology-c nimbus.host=192.168.0.107

13. View storm's UI monitoring interface (nimbus.host:7070)

This is the end of "installation and deployment of Storm On Yarn". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.