Shulou(Shulou.com)05/31 Report--

Editor to share with you what the HBase system architecture is, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

HBase system architecture diagram

Component description

Client:

Use HBase RPC mechanism to communicate with HMaster and HRegionServer

Client communicates with HMaster for management operations

Client and HRegionServer perform data read and write operations

Zookeeper:

Zookeeper Quorum storage-ROOT- table address, HMaster address

HRegionServer registers itself with Zookeeper as Ephedral, and HMaster is aware of the health status of each HRegionServer at any time.

Zookeeper avoids HMaster single Point problem

HMaster:

There is no single point problem with HMaster. Multiple HMaster can be started in HBase, and a Master is always running through the Master Election mechanism of Zookeeper.

Mainly responsible for the management of Table and Region

1 manage the operation of adding, deleting, changing and checking the table by users

2 manage the load balance of HRegionServer and adjust the Region distribution

Responsible for the distribution of new Region after 3 Region Split

4 responsible for Region migration on failed HRegionServer after downtime of HRegionServer

HRegionServer:

The core module in HBase, which is mainly responsible for reading and writing data to the HDFS file system in response to the user Imax O request.

Client writes-> stores MemStore until MemStore fullness-> Flush becomes a StoreFile, until it grows to a certain threshold-> triggers Compact merge operation-> multiple StoreFile merges into one StoreFile, and carries out version merging and data deletion at the same time-> when the StoreFiles Compact gradually forms larger and larger StoreFile-> a single StoreFile size exceeds a certain threshold, the Split operation is triggered, and the current Region Split into two Region,Region will be offline. The Region of the two children from the new Split will be assigned to the corresponding HRegionServer by the HMaster, so that the pressure of the original one Region can be diverted to the two Region.

From this process, we can see that HBase only adds data, updates and deletes all the operations are done in the Compact phase, so the user write operation only needs to enter the memory to return immediately, so as to ensure the high performance of Compact.

HLog

Reasons for introducing HLog:

In a distributed system environment, system errors or downtime cannot be avoided. Once HRegionServer exits unexpectedly, the memory data in MemStore will be lost. The introduction of HLog is to prevent this situation.

Working mechanism:

There is a HLog object in every HRegionServer. HLog is a class that implements Write Ahead Log. Every time a user writes a Memstore, it also writes a piece of data to the HLog file. The HLog file scrolls out periodically and deletes the old file (data that has been persisted to the StoreFile). When the HRegionServer terminates unexpectedly, HMaster perceives it through Zookeeper. HMaster first processes the legacy HLog files, splits the log data of different region and places them in the corresponding region directory, and then redistributes the invalid region. When you get the HRegionServer of these region, you will find that there is a historical HLog to deal with in the process of Load Region, so the data in Replay HLog will be sent to MemStore, and then flush to StoreFiles to complete data recovery.

HBase storage format

All data files in HBase are stored on the Hadoop HDFS file system in two main formats:

The storage format of KeyValue data in 1 HFile HBase. HFile is the binary format file of Hadoop. In fact, StoreFile makes a lightweight package for HFile, that is, the bottom layer of StoreFile is HFile.

2 the storage format of WAL (Write Ahead Log) in HLog File,HBase, which is physically the Sequence File of Hadoop

HFile

KeyLength and ValueLength: two fixed lengths, representing the lengths of Key and Value, respectively

Key: Row Length is a fixed-length value, indicating the length of RowKey, and Row is RowKey.

Column Family Length is a fixed-length numeric value that represents the length of the Family

Then there is Column Family, then Qualifier, then two fixed-length numbers for Time Stamp and Key Type (Put/Delete)

The Value part doesn't have such a complex structure, it's pure binary data.

HLog File

A HLog file is an ordinary Hadoop Sequence File,Sequence File Key is a HLogKey object, and the HLogKey records the attribution information of the written data, in addition to the names of table and region, but also includes that sequence number and timestamp,timestamp are "write time", the starting value of sequence number is 0, or the last time it is stored in the file system sequence number.

The Value of HLog Sequece File is the KeyValue object of HBase, that is, the KeyValue in the corresponding HFile

The above is all the content of this article "what is the architecture of HBase system?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.