Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to use Storm to write a Crawler tool", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use Storm to write a Crawler tool"!

Reading background:

1: you may need to know the basics of crawlers.

2: you may need to have a preliminary understanding of the HTTP protocol.

3: you may need to have a preliminary understanding of the logic of Storm computing.

Package com.digitalpebble.storm.crawler;import backtype.storm.Config;import backtype.storm.LocalCluster;import backtype.storm.StormSubmitter;import backtype.storm.topology.TopologyBuilder;import backtype.storm.tuple.Fields;import com.digitalpebble.storm.crawler.bolt.IPResolutionBolt;import com.digitalpebble.storm.crawler.bolt.indexing.IndexerBolt;import com.digitalpebble.storm.crawler.bolt.parser.ParserBolt;import com.digitalpebble.storm.crawler.fetcher.Fetcher;import com.digitalpebble.storm.crawler.spout.RandomURLSpout / * * topology * / public class CrawlTopology {public static void main (String [] args) throws Exception {TopologyBuilder builder = new TopologyBuilder (); builder.setSpout ("spout", new RandomURLSpout ()); builder.setBolt ("ip", new IPResolutionBolt ()) .shuffleGrouping ("spout") Builder.setBolt ("fetch", new Fetcher ()) .fieldsGrouping ("ip", new Fields ("ip")); builder.setBolt ("parse", new ParserBolt ()) .shuffleGrouping ("fetch"); builder.setBolt ("index", new IndexerBolt ()) .shuffleGrouping ("parse"); Config conf = new Config () Conf.setDebug (true); conf.registerMetricsConsumer (DebugMetricConsumer.class); if (args! = null & & args.length > 0) {conf.setNumWorkers (3); StormSubmitter.submitTopology (args [0], conf, builder.createTopology ()) } else {conf.setMaxTaskParallelism (3); LocalCluster cluster = new LocalCluster (); cluster.submitTopology ("crawl", conf, builder.createTopology ()); Thread.sleep (10000); cluster.shutdown ();}

Thank you for your reading, the above is the content of "how to use Storm to write a Crawler tool". After the study of this article, I believe you have a deeper understanding of how to use Storm to write a Crawler tool. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.