Shulou(Shulou.com)06/01 Report--

Lu Chunli's work notes, who said that programmers should not have literary style?

Environment:

Hadoop-2.6.0

Hbase-1.0.1

Zookeeper-3.4.6

1. Brief process of Hadoop cluster configuration

2. Brief process of Zookeeper cluster configuration

3. Brief process of HBase cluster configuration

4. HBase as an input source example

View the data in the current hbase table m_domain

[hadoop@dnode1 conf] $hbase shellHBase Shell Enter 'help' for list of supported commands.Type "exit" to leave the HBase ShellVersion 1.0.1, r66a93c09df3b12ff7b86c39bc8475c60e15af82d, Fri Apr 17 22:14:06 PDT 2015hbase (main): 001to leave the HBase ShellVersion 0 > listTABLE m_domaint_domain2 row (s) in 0.9270 seconds= > ["m_domain", "t_domain"] hbase (main): 002 m_domain 0 > scan' m_domain'ROW COLUMN+CELL alibaba.com_19990415_20220523 column=cf:access_server, timestamp=1440947490018 Value=\ xE6\ x9D\ xAD\ xE5\ xB7\ x9Ealibaba.com_19990415_20220523 column=cf:exp_date, timestamp=1440947490018, value=2022\ xE5\ xB9\ xB405\ xE6\ x9C\ x8823\ xE6\ x97\ xA5alibaba.com_19990415_20220523 column=cf:ipstr, timestamp=1440947490018, value=205.204.101.42alibaba.com_19990415_20220523 column=cf:owner, timestamp=1440947490018, value=Hangzhou Alibaba Advertising Co.alibaba.com_19990415_20220523 column=cf:reg_date, timestamp=1440947490018 Value=1999\ xE5\ xB9\ xB404\ xE6\ x9C\ x8815\ xE6\ x97\ xA5baidu.com_19991011_20151011 column=cf:access_server, timestamp=1440947489956, value=\ xE5\ x8C\ x97\ xE4\ xBA\ xACbaidu.com_19991011_20151011 column=cf:exp_date, timestamp=1440947489956, value=2015\ xE5\ xB9\ xB410\ xE6\ x8811\ xE6\ x97\ xA5baidu.com_19991011_20151011 column=cf:ipstr, timestamp=1440947489956, value=220.181.57.217baidu.com_19991011_20151011 column=cf:reg_date, timestamp=1440947489956 Value=1999\ xE5\ xB9\ xB410\ xE6\ x9C\ x8811\ xE6\ x97\ xA52 row (s) in 1.4560 secondshbase (main): 003xB9 0 > quit

Implement the Mapper side

Package com.invic.mapreduce.hbase.source;import java.io.IOException;import java.util.Map;import java.util.Map.Entry;import java.util.NavigableMap;import java.util.Set;import org.apache.hadoop.hbase.Cell;import org.apache.hadoop.hbase.CellUtil;import org.apache.hadoop.hbase.client.Result;import org.apache.hadoop.hbase.io.ImmutableBytesWritable;import org.apache.hadoop.hbase.mapreduce.TableMapper;import org.apache.hadoop.hbase.util.Bytes Import org.apache.hadoop.io.Text;import org.apache.hadoop.io.Writable;/** @ author lucl * TableMapper extends from the Mapper class, and all Mapper classes with HBase as the input source need to inherit this class * / public class HBaseReaderMapper extends TableMapper {private Text key = new Text (); private Text value = new Text () @ Override protected void setup (Context context) throws IOException, InterruptedException {super.setup (context) } @ Override protected void map (ImmutableBytesWritable row, Result result,Context context) throws IOException, InterruptedException {/ / can explicitly give family {NavigableMap map = result.getFamilyMap ("cf" .getBytes ()); Set values = map.entrySet () For (Entry entry: values) {String columnQualifier = new String (entry.getKey ()); String cellValue = new String (entry.getValue ()); System.out.println (columnQualifier + "\ t" + cellValue) / /} / / there are multiple column families or uncertain column family names {String rowKey = new String (row.get ()); byte [] columnFamily = null Byte [] columnQualifier = null; byte [] cellValue = null; StringBuffer sbf = new StringBuffer (1024); for (Cell cell: result.listCells ()) {columnFamily = CellUtil.cloneFamily (cell) ColumnQualifier = CellUtil.cloneQualifier (cell); cellValue = CellUtil.cloneValue (cell); sbf.append (Bytes.toString (columnFamily)); sbf.append (".") Sbf.append (Bytes.toString (columnQualifier)); sbf.append (":"); sbf.append (new String (cellValue, "UTF-8"));} key.set (rowKey) Value.set (sbf.toString ()); context.write (key, value);} @ Override protected void cleanup (Context context) throws IOException, InterruptedException {super.cleanup (context);}}

Driver class that implements MapReduce

Package com.invic.mapreduce.hbase.source;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.client.Scan;import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;import org.apache.hadoop.hbase.util.Bytes;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat Import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner / * @ author lucl * HBase as an input source example * * / public class HBaseASDataSourceDriver extends Configured implements Tool {/ * @ param args * @ throws Exception * / public static void main (String [] args) throws Exception {/ / System.setProperty ("hadoop.home.dir") "E:\\ hadoop-2.6.0\\ hadoop-2.6.0\") Int exit = ToolRunner.run (new HBaseASDataSourceDriver (), args); System.out.println ("receive exit:" + exit);} @ Override public int run (String [] args) throws Exception {Configuration conf = HBaseConfiguration.create () / / Parameter configuration of hadoop / * conf.set ("fs.defaultFS", "hdfs://cluster"); conf.set ("dfs.nameservices", "cluster"); conf.set ("dfs.ha.namenodes.cluster", "nn1,nn2") Conf.set ("dfs.namenode.rpc-address.cluster.nn1", "nnode:8020"); conf.set ("dfs.namenode.rpc-address.cluster.nn2", "dnode1:8020"); conf.set ("dfs.client.failover.proxy.provider.cluster", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider") * / / hbase master / / property "hbase.master" has been deprecated since 0.90 / / Just passing the ZK configuration makes your client auto-discover the master / / conf.set ("hbase.master", "nnode:60000"); / / zookeeper quorum getConf () .set ("hbase.zookeeper.property.clientport", "2181") GetConf () .set ("hbase.zookeeper.quorum", "nnode,dnode1,dnode2"); / / whether to enable speculative execution mechanism getConf () .setBoolean ("mapreduce.map.speculative", false) for Map Task; / / whether to enable speculative execution mechanism getConf () .setBoolean ("mapreduce.reduce.speculative", false) for Reduce Task Job job = Job.getInstance (conf); job.setJobName ("MyBaseReaderFromHBase"); job.setJarByClass (HBaseASDataSourceDriver.class); job.setOutputFormatClass (TextOutputFormat.class) / * when reading data from HBase, the data is passed to the Mapper defined below, and the data is processed in the Mapper class * since the Reducer class is not specified in job, the default Reducer class is called to write the output of Mapper intact. * if you need to do some other separate processing in Reducer, you can customize the Reducer class to do some more processing. * / Scan scan = new Scan (); / / scan.addFamily (family); / / scan.addColumn (family, qualifier); byte [] tableName = Bytes.toBytes ("m_domain") TableMapReduceUtil.initTableMapperJob (tableName, scan, HBaseReaderMapper.class, Text.class, Text.class, job); Path path = new Path ("/" + System.currentTimeMillis ()); FileOutputFormat.setOutputPath (job, path); return job.waitForCompletion (true)? 0: 1;}}

View the results:

Problem record:

a. The implementation of the Times error through Eclipse, but did not analyze the reason

b. If the runtime Mapper class is defined in the Driver class in a cluster environment, an error will be reported.

ClassNotFound for HBaseASDataSourceDriver$HBaseReaderMapper init ()

C. the zookeeper connector always shows the connected 127.0.0.1 rather than the configured zookeeper.quorum

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.