Shulou(Shulou.com)05/31 Report--

This article mainly introduces "detailed explanation of the usage of Replication in HBase". In daily operation, I believe that many people have doubts about the detailed interpretation of the use of Replication in HBase. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts of "detailed explanation of the usage of Replication in HBase"! Next, please follow the editor to study!

Replication: replication, which refers to continuously copying the same data to multiple places for storage, is a common and important concept in various storage systems. It can refer to the replication of master and slave databases in the database, the replication between multiple clusters in a distributed cluster, and the replication between multiple replicas in a distributed system. The difficulty is that the data is usually constantly changing, and it is necessary to continuously reflect the changes to multiple copies of the data and ensure that these copies are completely consistent.

In general, copying data to multiple copies has the following benefits:

Multiple backups improve the reliability of data

Separate OLTP and OLAP requests through replication between master and slave databases / master / slave clusters

Improve availability, even if a single copy is dead, there can still be other copies to provide read and write services

Scalable to serve more read and write requests by adding copies

Replication between cross-regional data centers, Client reduces request latency by reading and writing to the nearest data center

Replication in HBase refers to replication between master and slave clusters, which is used to copy write records from the master cluster to the slave cluster. HBase currently supports three kinds of Replication, namely, asynchronous Replication, serial Replication and synchronous Replication.

Asynchronous Replication

If you want to figure out the Replication of HBase, you first need to understand the architecture of HBase. The HBase cluster is composed of a group of processes, which are divided into Master and RegionServer according to their roles, in which Master is responsible for DDL operations, such as creating and deleting tables, and RegionServer is responsible for DML operations, such as data read and write operations. From the data view, the Table in HBase will be divided into multiple Region according to Range, and then different RegionServer will be responsible for providing services.

The interior of RegionServer is mainly composed of BlockCache,MemStore and WAL. It should be noted that each Column Family of each Region has its own MemStore, but BlockCache and WAL are shared by multiple Region. WAL (Write-ahead logging) is a common technology in the database. All changes need to be persisted to the WAL before being written to the database, thus ensuring that the successfully written data can be played back from the WAL in the event of a failure.

Replication in HBase is also based on WAL, in which a thread called ReplicationSource is responsible for Replication within each RegionServer process in the main cluster, and a ReplicationSink thread is responsible for receiving Replication data within each RegionServer in the slave cluster. ReplicationSource records the WAL queues that need to be synchronized, and then constantly reads the contents of the WAL. At the same time, you can do some filtering according to the configuration of Replication, such as whether to copy the data of this table, and then call the Rpc to send to the RegionServer of the slave cluster. The ReplicationSink thread of the slave cluster is responsible for converting the received data into put/delete operations and writing it to the slave cluster in the form of batch.

Because the background thread reads the WAL asynchronously and copies it to the slave cluster, this Replication method is called asynchronous Replication. Normally, the latency for the slave cluster to receive the latest written data is at the second level.

Serial Replication

Serial Replication means that for a Region, it is copied to the standby cluster strictly according to the write order of the primary cluster, which is a special Replication. At the same time, the default asynchronous Replication is not serial, the main reason is that Region can be moved, for example, HBase moves Region when load balancing. Suppose the RegionA is first on the RegionServer1, and then it is moved to the RegionServer2. Due to the latency of the asynchronous Replication, the last part of the write record of the RegionA has not been fully replicated to the standby cluster. After Region moves to RegionServer2, it begins to receive new write requests, and RegionServer2 replicates to the standby cluster, so at this time RegionServer1 and RegionServer2 replicate to the standby cluster at the same time, and the order in which the write records are replicated to the slave cluster is uncertain.

In this extreme case, as shown in the figure above, it will also lead to inconsistency between the master and slave cluster data. For example, the last unsynchronized write operation on RegionServer1 is Put, while the first write operation in which RegionA is moved to RegionServer2 is Delete. On the main cluster, the write order is first Put and then Delete. If the Delete operation on RegionServer2 is copied to the standby cluster first, and then the slave cluster does Major compaction once, it will delete the Delete marker, and then the Put operation will be synchronized to the standby cluster. Because Delete has been dropped by Major compact, the Put will never be deleted. So the standby cluster will have more data than the main cluster.

The key to solving this problem is to ensure that new writes on the RegionServer2 must be replicated after the write operations on the RegionServer1 are replicated. So serial Replication introduces a concept called Barrier. Every time you Region open, a new Barrier is written, and its value is the maximum SequenceId read in Region open plus 1. SequenceId is an important concept in HBase. Each Region has a SequenceId, which is strictly incremented with data writes, and SequenceId is written to the WAL with each write operation. So when the Region moves, the Region will be reopened in the new RegionServer, and after a new Barrier,Region has been moved multiple times, multiple Barrier will be written to divide the write operation of the Region into multiple intervals. At the same time, each Region maintains a lastPushedSequenceId, which represents the SequenceId of the last successful write operation currently pushed by the Region, so that you can determine whether a write operation in the WAL can be replicated to the standby cluster according to the Barrier list and lastPushedSequenceId.

As an example in the above figure, the write record of Pending needs to wait for lastPushedSequenceId to push to Barrier2 before it can be replicated. Since there is only one RegionServer between each interval to replicate, only the RegionServer in the same interval as the lastPushedSequenceId can be replicated, and the lastPushedSequenceId can be updated continuously after the replication is successful, while the RegionServer in each interval after the lastPushedSequenceId needs to wait for the lastPushedSequenceId to be pushed to the beginning of the interval Barrier before starting replication, thus ensuring that the write operation of the Region can be serially replicated to the slave cluster in strict accordance with the write order of the master cluster.

Synchronous Replication

Synchronous Replication is the concept of symmetry with asynchronous Replication, which means that writes to the primary cluster must be written synchronously to the standby cluster. The biggest problem with asynchronous Replication is that there is a delay in replication, so when the master cluster and the whole cluster hang up, the slave cluster does not have the complete data that has been written. For businesses with high consistency requirements, it is not possible to completely cut the read and write to the slave cluster, because at this time, some recently written data may not be read from the slave cluster. Therefore, the core idea of synchronous Replication is to write a RemoteWAL to the slave cluster while writing to the master cluster WAL. Only if the WAL of the master cluster and the RemoteWAL of the slave cluster are successfully written, will it be returned to Client to say that the write is successful. In this way, when the master cluster dies, all the write records on the master cluster can be played back and forth according to Remote WAL on the slave cluster, thus ensuring the consistency of the data between the slave cluster and the master cluster.

It is important to note that synchronous Replication is based on asynchronous Replication, which means that the replication link of asynchronous Replication remains, while a new step is added to write Remote WAL. For the specific implementation details, the first is to add a concept of Sync replication state, which has a total of three states, namely Active,Downgrade Active and Standby. The transition relationship between these states is shown in the following figure. Standby needs to be promoted to Downgrade Active before it can be promoted to Active. But Active can be downgraded to Standby directly. This state is currently stored in ReplicationPeerConfig, which indicates which state a cluster is in in this ReplicationPeer.

Then a DualAsyncFSWAL is implemented to write both the WAL of the primary cluster and the Remote WAL of the backup cluster. The operation of writing WAL is for the rpc request of HDFS, which has three kinds of results: success, failure, or timeout. When the timeout occurs, the result is uncertain for HBase, that is, the data may or may not have been successfully written to WAL or Remote WAL. Only when writing succeeds at the same time or fails at the same time will the master cluster and slave cluster have the same WAL. If the master cluster succeeds in writing WAL, fails to write Remote WAL or times out, then the data in the master cluster WAL may be more than the Remote WAL in the slave cluster. On the contrary, if the Remote WAL of the slave cluster is successful, and the WAL of the primary cluster fails or times out, there may be more data in the Remote WAL of the slave cluster than that of the primary cluster. When both sides time out, I'm not sure there's too much over there. Therefore, the key to synchronous replication is how to ensure the ultimate consistency of the master and standby cluster data in the above cases. That is, when switching between master and slave clusters, Client should always see consistent data from the master / slave clusters. And when the master / slave does not reach a consistent intermediate state, some restrictions are needed to ensure that the Client cannot read the result of such intermediate inconsistency. So to sum up, the master and slave clusters are ultimately consistent, but for Client, it is strongly consistent, that is, the data successfully written must be read by both the master and slave clusters. For specific implementation details, please refer to HBaseCon Asia 2018:HBase at Xiaomi [1].

Compared with asynchronous replication, synchronous replication mainly affects the write path, and from our test results, there will be about 14% performance degradation, which is planned to be optimized in HBASE-20422 [2].

Custom Replication Endpoint

In addition to the above three Replication, HBase also supports plug-in Replication Endpoint, and you can customize Replication Endpoint to implement a variety of functions. As far as Xiaomi is concerned, we have implemented Replication Endpoint between different tables, for example, table An of the main cluster can be copied to table B of the slave cluster, and its application scenarios include cluster migration, Namespace replacement or table name change. At the same time, in order to implement Point-in-time Recovery, we have made a Replication Endpoint that can synchronize data to Xiaomi message queue Talos. When there is a scenario that needs to recover T1 at a certain point in time, we can first find the Snapshot closest to T1 in the cold backup for recovery, and then put the data in the message queue back and forth to T1 time point to achieve Point-in-time Recovery.

At this point, the study of "detailed explanation of the use of Replication in HBase" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.