Shulou(Shulou.com)05/31 Report--

This article mainly explains "Storm how to write a crawler", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "Storm how to write a crawler" bar!

Package com.digitalpebble.storm.crawler.bolt.indexing;import java.util.Map;import org.slf4j.LoggerFactory;import backtype.storm.task.OutputCollector;import backtype.storm.task.TopologyContext;import backtype.storm.topology.OutputFieldsDeclarer;import backtype.storm.topology.base.BaseRichBolt;import backtype.storm.tuple.Tuple;import com.digitalpebble.storm.crawler.StormConfiguration;import com.digitalpebble.storm.crawler.util.Configuration;/** * A generic bolt for indexing documents which determines which endpoint to use * based on the configuration and delegates the indexing to it. * * / @ SuppressWarnings ("serial") public class IndexerBolt extends BaseRichBolt {private Configuration config; private BaseRichBolt endpoint; private static final org.slf4j.Logger LOG = LoggerFactory .getLogger (IndexerBolt.class); public void prepare (Map conf, TopologyContext context, OutputCollector collector) {config = StormConfiguration.create (); / / get the implementation to use / / and instanciate it String className = config.get ("stormcrawler.indexer.class") If (className = = null) {throw new RuntimeException ("No configuration found for indexing");} try {final Class implClass = (Class) Class .forName (className); endpoint = implClass.newInstance ();} catch (final Exception e) {throw new RuntimeException ("Couldn't create" + className, e) } if (endpoint! = null) endpoint.prepare (conf, context, collector);} public void execute (Tuple tuple) {if (endpoint! = null) endpoint.execute (tuple);} public void declareOutputFields (OutputFieldsDeclarer declarer) {if (endpoint! = null) endpoint.declareOutputFields (declarer) }} Thank you for your reading, the above is the content of "how to write a crawler in Storm". After the study of this article, I believe you have a deeper understanding of how to write a crawler in Storm, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.