Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to install storm eclipse and debug TopN instances in windows. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

One: install JD

Configure the values of Java environment variables JAVA_HOME, Path and CLASSPATH as (set according to your installation status, for reference here):

D:\ java\ jdk1.8

% JAVA_HOME%/bin;%JAVA_HOME%/jre/bin

.;% JAVA_HOME%/lib/dt.jar;%JAVA_HOME%/lib/tools.jar (to add. Represents the current path)

Two: install Python

To test the installation effect, we will deploy the word coun program in the storm-starter project case, using multi-lang bolt written by python, using python 2.7.11, and the installation path is at:

C:\ Python27\

Three: install and run ZooKeeper

Download Apache Zookeeper 3.4.8, decompression configuration:

> cd zookeeper-3.4.8

> copy conf\ zoo_sample.cfg conf\ zoo.cfg

>.\ bin\ zkServer.cmd

Four: install Storm

The official windows version of Storm has not been released yet, here. Download, source code here download.

Note 1:

Be sure to use this version of the source code, otherwise startup will report a variety of errors that have nothing to do with jdk, python, zookeeper, or eclipse versions.

Http://dl.dropboxusercontent.com/s/iglqz73chkul1tu/storm-0.9.1-incubating-SNAPSHOT-12182013.zip

Configure Storm environment variables

Storm requires STORM_HOME and JAVA_HOME. For example, STORM_HOME is:

C:\ storm-0.9.1-incubating-SNAPSHOT-12182013\

Add the following to PATH:

% STORM_HOME%\ bin;C:\ Python27\ Lib\ site-packages\; C:\ Python27\ Scripts\

This is slightly different from the reference article, and the following figure is the configuration given in the reference article.

JAVA_HOME has manually configured the environment variables when installing JDK, while Python seems to configure the environment variables automatically by default

There is no Scripts folder in my Python directory, so I can configure it for the time being and will not affect the use below.

Five: start Nimbus, Supervisor, and Storm UI Daemons

Nimbus

Note 2:

Be sure to execute subsequent commands in the STORM_HOME directory, or an error will be reported.

ERROR backtype.storm.event-Error when processing event

Java.lang.RuntimeException: java.io.InvalidClassException: clojure.lang.APersistentMap; local class incompatible: stream classdesc serialVersionUID = 8648225932767613808, local class serialVersionUID = 270281984708184947

At backtype.storm.utils.Utils.deserialize (Utils.java:86) ~ [storm-core-0.9.1-incubating-SNAPSHOT-12182013.jar:na]

> cd STORM_HOME%

> storm nimbus

Supervisor

> cd STORM_HOME%

> storm supervisor

Storm UI # is optional, or you can view all storm tasks with storm list

> cd STORM_HOME%

> storm ui

The browser opens http://localhost:8080/ to see Storm running.

Six: deploy Word count

Download download a pre-built jar.

Deploy the jar locally:

Storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology WordCount-c nimbus.host=localhost

If you refresh the Storm UI page, you will see a list of "WordCount" topology. Click the link to confirm that it processes the data.

Seven: eclipse debug TopN instance

Storm asks for the topN of the password in the csdn password library, and directly debugs and runs it in eclipse:

Package com.bj.test.top10;/** * @ Author:tester * @ DateTime: 7:58:45 * @ Description: Spout on June 21, 2016 as a data source, it implements the IRichSpout interface, which reads a text file and sends each line of its contents to bolt. * @ Version:1.0*/import java.io.BufferedReader;import java.io.FileNotFoundException;import java.io.FileReader;import java.util.Map;import backtype.storm.spout.SpoutOutputCollector;import backtype.storm.task.TopologyContext;import backtype.storm.topology.OutputFieldsDeclarer;import backtype.storm.topology.base.BaseRichSpout;import backtype.storm.tuple.Fields;import backtype.storm.tuple.Values;public class PasswdSpout extends BaseRichSpout {private SpoutOutputCollector collector; private FileReader fileReader; private boolean completed = false Public void ack (Object msgId) {System.out.println ("= OK:" + msgId);} public void close () {} public void fail (Object msgId) {System.out.println ("+ FAIL:" + msgId) } / * this is the main method of Spout, where we read the text file and emit each line of it (to bolt) * this method will be called again and again in order to reduce its consumption of CPU Let it sleep * * / public void nextTuple () {/ * The nextuple it is called forever when the task is finished. So if we have been readed the file * we will wait and then return * / if (completed) {try {Thread.sleep (1000) } catch (InterruptedException e) {/ / Do nothing} return;} String line; / / Open the reader BufferedReader reader = new BufferedReader (fileReader) Try {/ / Read all lines while ((line = reader.readLine ())! = null) {String [] words = line.split ("#"); String passwd = words [1] .trim () / / Emit the word collector.emit (new Values (passwd)); / * for (String word: words) {word = word.trim (); if (! word.isEmpty ()) {word = word.toLowerCase () / / Emit the word collector.emit (new Values (word));}} * /}} catch (Exception e) {throw new RuntimeException ("Error reading tuple", e) } finally {completed = true }} / * this is the first method, which receives three parameters, the first is the configuration when creating the Topology, and the second is all the Topology data. The third is used to transmit Spout data to bolt * * / public void open (Map conf, TopologyContext context, SpoutOutputCollector collector) {try {/ / get the file path this.fileReader = new FileReader (conf.get ("wordsFile"). ToString () specified when creating Topology. } catch (FileNotFoundException e) {throw new RuntimeException ("Error reading file [" + conf.get ("wordFile") + "]");} / / initialize emitter this.collector = collector } / * * Declare the output field "word" * / public void declareOutputFields (OutputFieldsDeclarer declarer) {declarer.declare (new Fields ("word")) }} / package com.bj.test.top10;import java.util.HashMap;import java.util.Map;import java.util.NavigableMap Import java.util.TreeMap;import backtype.storm.task.TopologyContext;import backtype.storm.topology.BasicOutputCollector;import backtype.storm.topology.OutputFieldsDeclarer;import backtype.storm.topology.base.BaseBasicBolt;import backtype.storm.tuple.Tuple;import static com.bj.test.top10.SortMapByValue.*;public class Top10Bolt extends BaseBasicBolt {Integer id; String name; NavigableMap counters / * * the cleanup work performed by Topology, such as closing the connection and releasing resources, is written here * because this is only a Demo, we use it to print our counter * * / @ Override public void cleanup () {System.out.println ("> Word Counter [" + name+ "-" + id+ "]

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.