How to write and store HDFS file content data into HBase

This article mainly explains "how to write HDFS file content data to HBase", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to write HDFS file content data into HBase" bar!

Write and store HDFS file content data in HBase:

For some large files, need to be stored in HBase, the idea is to first transfer the file to HDFS, using map stage to read the pair, you can upload these key-value pairs to HBase in reduce.

It is already fixed to specify a file in HDFS, and then write these key-value pairs to HBase in reduce.

Public class HBaseAndMapReduce3 {public static void main (String [] args) throws Exception {System.exit (run ());} public static int run () throws Exception {Configuration conf = new Configuration (); conf = HBaseConfiguration.create (conf); conf.set ("hbase.zookeeper.quorum", "192.168.226.129") Job job = Job.getInstance (conf, "findFriend"); job.setJarByClass (HBaseAndMapReduce3.class); job.setInputFormatClass (KeyValueTextInputFormat.class); job.setMapOutputKeyClass (Text.class); job.setMapOutputValueClass (Text.class); DateFormat df = new SimpleDateFormat ("yyyyMMddHHmmssS") FileInputFormat.addInputPath (job, new Path ("hdfs://192.168.226.129:9000/hbasemapreduce1/2016051818564427/part-r-00000")); / / write data to Hbase database TableMapReduceUtil.initTableReducerJob ("friend", FindFriendReducer.class, job); checkTable (conf) Return job.waitForCompletion (true)? 0: 1;} private static void checkTable (Configuration conf) throws Exception {Connection con = ConnectionFactory.createConnection (conf); Admin admin = con.getAdmin (); TableName tn = TableName.valueOf ("friend") If (! admin.tableExists (tn)) {HTableDescriptor htd = new HTableDescriptor (tn); HColumnDescriptor hcd = new HColumnDescriptor ("person"); htd.addFamily (hcd); admin.createTable (htd) System.out.println ("Table does not exist, new table created successfully") }} public static class FindFriendReducer extends TableReducer {@ Override protected void reduce (Text key, Iterable values) Reducer.Context context) throws IOException, InterruptedException {Put put = new Put (key.getBytes ()) Put.addColumn (Bytes.toBytes ("person"), Bytes.toBytes ("nickname"), values.iterator (). Next (). GetBytes (); context.write (new ImmutableBytesWritable (key.getBytes ()), put);}

/ / the contents of the original data file:

Hadoop Berg-OSChina,BergBerghbase OSChina,BergBergzookeeper OSChina,BergBerg

/ / store the contents of the files in HDFS into HBase, and know through the client's full table scan:

Hbase (main): 003friend'ROW COLUMN+CELL hadoop column=person:nickname 0 > scan 'friend'ROW COLUMN+CELL hadoop column=person:nickname, timestamp=1463748372584, value=Berg-OSChina,BergBerg hbasep column=person:nickname, timestamp=1463748372584, value=OSChina BergBerggBerg zookeeper column=person:nickname, timestamp=1463748372584, value=OSChina,BergBerggBerg 3 row (s) in 0.2850 seconds Thank you for reading The above is the content of "how to write HDFS file content data to HBase". After the study of this article, I believe you have a deeper understanding of how to write HDFS file content data to HBase. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!