Shulou(Shulou.com)05/31 Report--

Editor to share with you how Mapreduce scans the hbase table to build a solr index. I hope you will get something after reading this article. Let's discuss it together.

Package com.hbase.index;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.mapreduce.Counter;import org.apache.hadoop.mapreduce.Job;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public class RebuildHbaseIndex {public static final Logger LOG = LoggerFactory .getLogger (RebuildHbaseIndex.class) Public static void main (String [] args) throws IOException,ClassNotFoundException, InterruptedException {Configuration conf = HBaseConfiguration.create (); conf.setBoolean ("mapred.map.tasks.speculative.execution", false); / / conf.setInt ("hbase.client.scanner.caching", 100) at a time; String [] tbNames= {"Suggest"}; for (int item0) I = commitSize) {try {LOG.info ("add document: Adding" + Integer.toString (docs.size ()) + "documents"); solr.add (docs); / / Index document} catch (final SolrServerException e) {final IOException ioe = new IOException (); ioe.initCause (e); throw ioe } docs.clear ();} context.getCounter (Counters.ROWS) .increment (1) } / / call @ Override protected void cleanup (org.apache.hadoop.mapreduce.Mapper.Context context) throws IOException at the end of the task, InterruptedException {try {if (! docs.isEmpty ()) {LOG.info ("clear queue: Adding" + Integer.toString (docs.size ()) + "documents"); solr.add (docs) Docs.clear ();} catch (final SolrServerException e) {final IOException ioe=new IOException (); ioe.initCause (e); throw ioe;}} public static Job createSubmittableJob (Configuration conf, String tableName) throws IOException {Job job=Job.getInstance (conf, "SolrIndex_" + tableName); job.setJarByClass (SolrIndexerMapper.class) Scan scan=new Scan (); / / scan data is not put in cache, one-time scan.setCacheBlocks (false); job.setOutputFormatClass (NullOutputFormat.class); TableMapReduceUtil.initTableMapperJob (tableName, scan, SolrIndexerMapper.class, null, null, job); / / No output is required, key and value type is null job.setNumReduceTasks (0); / / No reduce task return job }} after reading this article, I believe you have a certain understanding of "how Mapreduce scans hbase tables to build solr index". If you want to know more about it, welcome to follow the industry information channel. Thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.