Shulou(Shulou.com)06/01 Report--

The main content of this article is "introduction to the storage format of hbase". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor to take you to learn the "introduction to the storage format of hbase"!

All data files in HBase are stored on the Hadoop HDFS file system, including the two file types proposed above:

1. HFile, the storage format of KeyValue data in HBase. HFile is the binary format file of Hadoop. In fact, StoreFile makes a lightweight package for HFile, that is, the underlying layer of StoreFile is HFile.

2. The storage format of WAL (Write Ahead Log) in HLog File,HBase, which is physically the Sequence File of Hadoop

HFile

The following figure shows the storage format of HFile:

First of all, the HFile file is indefinite in length, and there are only two pieces of fixed length: Trailer and FileInfo. As shown in the figure, there is a pointer in Trailer to the starting point of other data blocks. Some Meta information of the file is recorded in File Info, such as AVG_KEY_LEN, AVG_VALUE_LEN, LAST_KEY, COMPARATOR, MAX_SEQ_ID_KEY and so on. The Data Index and Meta Index blocks record the starting point of each Data block and Meta block.

Data Block is the basic unit of HBase HBase O. In order to improve efficiency, there is a Block Cache mechanism based on LRU in HRegionServer. The size of each Data block can be specified by parameters when creating a Table, large Block is good for sequential Scan, and small Block is good for random query. Each Data block is made up of pairs of KeyValue in addition to the Magic at the beginning, and the Magic content is a random number designed to prevent data corruption. The internal structure of each KeyValue pair is described in more detail later.

Each KeyValue pair in HFile is a simple byte array. But this byte array contains a lot of items and has a fixed structure. Let's take a look at the specific structure inside:

It begins with two fixed-length values that represent the length of the Key and the length of the Value. This is followed by Key, starting with a fixed-length number, indicating the length of RowKey, followed by RowKey, then a fixed-length value, representing the length of Family, then Family, then Qualifier, then two fixed-length values, representing Time Stamp and Key Type (Put/Delete). The Value part doesn't have such a complex structure, so it's pure binary data.

HLogFile

The above figure shows the structure of the HLog file, in fact, the HLog file is an ordinary Hadoop Sequence File,Sequence File Key is a HLogKey object, HLogKey records the attribution information of the written data, in addition to the names of table and region, but also includes that sequence number and timestamp,timestamp are "write time", the starting value of sequence number is 0, or the last time it is stored in the file system sequence number.

The Value of HLog Sequece File is the KeyValue object of HBase, that is, the KeyValue in the corresponding HFile, as described above.

At this point, I believe that everyone on the "introduction to the storage format of hbase" have a deeper understanding, might as well to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.