Shulou(Shulou.com)05/31 Report--

This article mainly talks about "what is the method of migrating Hbase data". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the method of Hbase data migration?"

Background

In the process of using Hbase, the Hbase cluster used often requires data migration for some reason. In most cases, you can negotiate with users to migrate offline, and it is easier to migrate offline data. The entire data storage directory of Hbase can be relocated, but when there is a large amount of data in the cluster, the file copy time is very long, and the impact on the customer's business is also long, which often cannot be completed in the time window given by the customer. This paper gives a migration idea. You can use the functions of Hbase to migrate the cluster and reduce the interruption time of the cluster business.

Brief introduction

As we all know, Hbase has the function of snapshot snapshot. Using the snapshot, you can record the data of the table at a certain point in time and save the snapshot. When needed, you can restore the table data to the way it was when you took the snapshot. We can export the full amount of data at a certain point in time by using Hbase's snapshot.

Because the user's business is still constantly writing to the table, in addition to migrating the full amount of data before the snapshot time point, we also need to migrate the continuous incremental data after the snapshot time point. Here, if we can use the method of double write, it would be nice to write the data to two clusters, but the user's business will not do so. If you do so, you have to ensure the transaction consistency of double writes. Therefore, you can make use of the replication feature of Hbase. The replication function itself retains the WAL log records of the source cluster to play back and write to the destination cluster, so that the user's business end-> original cluster-> destination cluster is a serial data stream, and Hbase ensures the correctness of the data.

So the method of this migration is to use snapshot to migrate all data and replication to migrate incremental data.

Migration steps

T0: configure the Replication relationship between the old cluster An and the new cluster B. the data of Replication is synchronized from cluster A to cluster B, and the table is set to be synchronized. From now on, the data newly written to cluster A will be retained in the WAL log.

T1: generate the full amount of data at this point in time, and export the data at this point in time to the new cluster B by creating and exporting snapshot data

T2: the new cluster B imports the snapshot data at T1 time, and a table is created from the snapshot in the new cluster B. at this time, the Replication relationship set on the old cluster A cluster will automatically start playing back the WAL logs retained at T0 time into the table of the new cluster B, and start incremental data synchronization.

T3: since the operation between T0-T3 takes a period of time, a lot of WAL log files will be accumulated at this time, and it will take a certain time to synchronize to the new cluster. Here, you need to monitor the data synchronization. When the old cluster WAL is gradually consumed, you can stop the write business of the old cluster and prepare to switch all the read and write business to the new cluster B.

T4: there should be a very short period of time between T3-T4, and there is only a certain interruption in the whole migration. At this point, the user is allowed to completely cut the business to the new cluster B, and the migration is completed.

Commands involved in the operation

1. Set the peer relationship between cluster An and cluster B

In the source cluster Hbase shell, set peer

Add_peer 'peer_name','ClusterB:2181:/Hbase'

two。 Set the replication property in the table of cluster A

If the target table is named Student, get the Family=f first.

Enter into Hbase shell

Alter 'Student', {NAME = >' fanciful narrative scope = >'1'}

3. Create a snapshot of the table in cluster A

In Hbase shell

Snapshot' Student','Student_table_snapshot'

4. Export snapshots in cluster A

Hbase org.apache.hadoop.Hbase.snapshot.ExportSnapshot-snapshot Student_table_snapshot-copy-to / snapshot-backup/Student

5. Place the snapshot data in the corresponding directory of cluster B

The above command exports two directories, one is snapshot metadata and the other is raw data

Put the metadata in / Hbase/.Hbase-snapshot and the original data in the / Hbase/archive directory

Since the archive directory of Hbase will be cleaned regularly, you can set the Hbase.master.cleaner.interval value of cluster B's master to a higher point in advance to avoid accidental data cleanup during the copy process.

If there is no corresponding directory in cluster B, you can create it in advance.

Hdfs dfs-mkdir-p / Hbase/.Hbase-snapshot

Hdfs dfs-mkdir-p / Hbase/archive/data/default/

Move the exported snapshot file to the snapshot directory

Hdfs dfs-mv / snapshot-backup/Student/.Hbase-snapshot/Student_table_snapshot / Hbase/.Hbase-snapshot/

Hdfs dfs-mv / snapshot-backup/Student/archive/data/default/Student / Hbase/archive/data/default/

6. Restore a snapshot of the table in the new cluster B

Enter Hbase shell

Restore_snapshot 'Student_table_snapshot'

After the recovery is complete, remember to adjust the value of Hbase.master.cleaner.interval in the hmaster of cluster B.

At this point, I believe you have a deeper understanding of "what is the migration method of Hbase data?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.