Shulou(Shulou.com)05/31 Report--

Editor to share with you how to achieve data migration between hbase clusters, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

HBase migrates data

First, you need to migrate when the hbase cluster stops.

Steps:

(1) execute Hadoop distcp-f filelist "hdfs://new cluster ip:9000/hbasetest"

(2) execute. / hbase org.jruby.Main add_table.rb / hbase/table20111222 in new cluster and add the copied table to .Meat. Medium (there will be an inconsistent number of region, which can only be solved by restarting hase)

Note: (1) filelist is a list file on hdfs, with the following contents:

/ hbase/table20111222

/ hbase/table20120131

(2) if the hadoop versions of the two clusters are not the same, you need to execute hadoop distcp on the new cluster, otherwise a read / write exception will occur.

Second, carry out data migration while the cluster is running

1 、 Replication:

This is a dynamic backup (can be understood as a real-time backup)

Steps:

(1) modify the table attributes that need to be migrated in old cluster:

Disable 'your_table'

Alter 'your_table', {NAME = >' family_name', REPLICATION_SCOPE = >'1'}

Enable 'your_table'

(2) Open the replication of the new cluster cluster and modify the hbase-site.xml

Hbase.replication

True

(3) add peer and execute it in the hbase shell of new cluster: add_peer '1launch replication and execute start_replication

Note: the hadoop versions of the two clusters need to be the same, otherwise there will be read and write errors

2 、 CopyTable:

You can copy a table in this cluster, or you can copy the table to another cluster.

Command:. / hbase org.apache.hadoop.hbase.mapreduce.CopyTable-- peer.adr=new cluster ip:2181:/hbase zy_test

Description: (1) the copy is completed, and there is no need to restart the machine. You can see the table in new cluster.

(2) Stability also needs to be considered. There is a problem encountered in the testing process. Lookedup root region location always stays here during the copy process. There are no errors and related information in the log.

3 、 Export and Import

Steps:

(1) execute:. / hbase org.apache.hadoop.hbase.mapreduce.Export test hdfs://new cluster ip:9000/zhuangyang/test on old cluster

(2) execute:. / hbase org.apache.hadoop.hbase.mapreduce.Import test hdfs://new cluster ip:9000/zhuangyang/test on new cluster

Description:

(1) be sure to write the whole path, not the relative road strength.

(2) before import, the table needs to be created in new cluster in advance.

All above are the ways to realize data migration when old cluster and new cluster networks are connected.

HBase migration data scenario 2 (two clusters cannot communicate)

It is easy to copy data and migrate HBase data between two clusters that can communicate directly.

But what if two clusters are on different networks and can't communicate?

A scheme was tested today:

To migrate a table (table1) from cluster 1 to cluster 2 (the two clusters are invisible to each other), follow these steps

1. Copy the table file of Cluster 1 to the local disk:

/ app/cloud/Hadoop/bin/hadoop fs-copyToLocal / hbase/table1 / data0/hbase-backup/table1

2. For file operation, it's very simple. You can copy it all you want.

3. If there is a corresponding table file in cluster 2, delete it and copy it

/ app/cloud/hadoop/bin/hadoop fs-rmr / hbase/table1

/ app/cloud/hadoop/bin/hadoop fs-copyFromLocal / data0/hbase-backup/table1 / hbase/table1

4. Reset the table in .meta. Partition information in the table

Bin/hbase org.jruby.Main bin/add_table.rb / hbase/table1

In addition:

1. What if the data in the table is too large? Then copy in batches according to the folder data of the table in HDFS.

2. What if two clusters can communicate with each other? The first scheme can be adopted.

These are all the contents of the article "how to achieve data migration between hbase clusters". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.