Shulou(Shulou.com)06/01 Report--

This article mainly explains the "Hbase data writing and data storage process", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "Hbase data writing and data storage process"!

The process of writing and storing data in Hbase

Client write-> store into MemStore until MemStore full-> Flush into a StoreFile, until grow to a certain threshold-> start Compact merge operation-> multiple StoreFile merge into one StoreFile at the same time for version merging and data deletion-> when the StoreFiles Compact gradually forms a larger and larger StoreFile-> a single StoreFile size exceeds a certain threshold, the Split operation is triggered, and the current Region Split into two Region,Region will be offline. The Region of two children from the new Split will be assigned to the corresponding HRegionServer by HMaster, so that the pressure of the original one Region can be diverted to the two Region. This process shows that HBase is just adding data, and any update and deletion operations are done in the Compact phase. Therefore, the user write operation only needs to enter the memory to immediately return, thus ensuring the high performance of HBase O.

A supplement to the above process:

Supplementary 1:HStore storage is the core of HBase storage, which consists of two parts, one is MemStore and the other is StoreFiles.

Supplement the functions of 2:HLog:

In a distributed system environment, system errors or downtime cannot be avoided, once exiting outside HRegionServer

Memory data in MemStore is lost, and HLog was introduced to prevent this.

How it works: there is a HLog object in every HRegionServer. HLog is a class that implements Write Ahead Log. Each user operation writes Memstore, a piece of data is also written to the HLog file, and the HLog file scrolls out regularly and deletes the old file (data that has been persisted to the StoreFile). When HRegionServer terminates unexpectedly, HMaster perceives it through Zookeeper. HMaster first processes the legacy HLog files, splits the log data of different region and places them in the corresponding region directory, and then redistributes the invalid region (with the newly split log). In the process of Load Region, the HRegionServer that receives these region will find that there is a historical HLog to deal with, so the data in Replay HLog will be transferred to MemStore, and then flush to StoreFiles to complete data recovery.

Supplementary 3:Region is made up of HFile in StoreFiles,StoreFiles, data block of hbase in Hfile, and many keyvalue pairs in a data block, and each keyvalue stores the values we need.

Supplement 4:

Let's take a look at the picture above:

A table, there are two column families (a red color, a yellow color), a column family has two columns, you can see from the figure, this is the biggest feature of the column database, the same column family of data together, we also found that if there are multiple versions, but also save multiple versions. Finally, we also found that there is such a value in it: R1: key value, cf1: the name of the family, C1: listed. T1: version number, value value (the last picture shows where the value value can be stored). From this point of view, we find that if we design the table with these things: R1: key value, cf1: the name of the column family, C1: if the listed name is shorter, we will save a lot of storage space!

Also, we should get the following understanding from this picture:

When we look at the penultimate picture, the efficiency of field filtering decreases significantly from left to right, so users can consider moving some important filtering information left to the appropriate location in the design of keyvalue, so as to improve query performance without changing the amount of data. To put it simply, users should try to store the query dimension or information in the row, because it is the most efficient to filter data.

After getting the above understanding, we should also have this awareness:

The data of HBase will be sequentially stored to a specific range, because we usually store it sequentially, so it will always be stored on the same region. Because a region can only be managed by one server, we always add it to the same region, which will cause read and write hotspots and degrade the performance of the cluster. Well, there is still a solution to this, and all I can think of is, for example, if we have nine servers, then we will go back to the current time, then touch 9 and add it to the row key prefix, so that it will be evenly distributed to different region servers. The advantage of this is that because the connected data is distributed to different servers, users can read data in parallel with multiple threads. This increases the throughput of the query.

For our version control, we either synchronize time on multiple servers or simply set a client timestamp when put inserts data. (because if we don't add it as shown, people will give us time to add to our own servers. )

Supplement 5:

When setting the meter, there are two ways of design, one is the high watch design, the other is the fat watch design. According to HBase's splitting rules, our high table design is easier to split (using combined keys), but if we design a fat table, and our fat data needs to be modified frequently, this design is reasonable, because our HBase ensures row-level atomicity, but if designed as a high table, it is not appropriate, because cross-row atomicity is not guaranteed.

Supplement 6:

Write caching

The operation of each put is actually an operation of RPC, which transmits the data from the client to the server and then returns, which is only suitable for operations with a small amount of data. If an application needs to store thousands of rows of data into the HBase table every second, this is not appropriate. HBase's API is equipped with a client-side write buffer that collects put operations, and then calls the RPC operation to send the put to the server at one time. By default, client buffers are prohibited. The buffer can be activated by automatically brushing to FALSE. Table.setAutoFlush (false); void flushCommits () throws IOException this method forces data to be written to the server. Users can also configure the size of the client write buffer according to the following methods. The default size of void setWritaeBufferSize (long writeBufferSize) throws IOException; is 2MB, which is also moderate. The average user inserts a small amount of data, but if the data you insert is large, you may want to consider increasing this value. This allows the client to more efficiently organize a certain amount of data into a set of RPC requests to execute. It is also troublesome to set a write buffer for each user's HTable. To avoid trouble, users can use the

Set a large default value for the user in Hbase-site.xml.

Hbase.client.write.buffer

20971520

Supplement 7:

Hbase supports a large number of algorithms and supports compression algorithms above the column family level. Unless there is a special reason, we should try to use compression, which usually brings better performance. Through some tests, we recommend using the SNAPPY algorithm to compress our hbase.

Hbase read data:

Client- > zookeeper- > .Root-> .meta-> the user data table zookeeper records the path information of .Root (root has only one region), the region information of .meta (.meta may have multiple region), and the information of region in .meta.

Supplement 1:

In HBase, all storage files are divided into small blocks that are loaded into memory during get or scan operations, similar to storage unit pages in RDBMS. The default size of this parameter is 64K. Set by the above way: void setBlocksize (int s); (the default size of Hfile in HBase is 64K has nothing to do with the block of HDFS is 64m) HBase sequentially reads a data block into the memory cache, and when it reads adjacent data, it can be read in memory without the need to read again from the disk, effectively reducing the number of disk Imando. This parameter defaults to TRUE, which means that each read block is cached in memory. However, if the user reads a specific column family sequentially, it is best to set this property to FALSE to disable the use of cache fast. The reason described above: if we access a specific column family, but we still enable this feature, our mechanism will load the data of other column families that we do not need into memory, adding to our burden. We use the condition that we get adjacent data. Void setBlockCacheEnabled (boolean blockCacheEnable)

Supplement 2:

1: automatic writing is prohibited.

When we have a large amount of data to insert, if we do not prohibit it, the Put instances will be transferred to the regio server one by one.

If the user disables automatic flushing, the put operation will not be sent until the write buffer is filled.

2: use scan cache.

If HBase is used as an input source for a mapreduce job, it is best to scan it as an mapreduce job input

The cache of the instance of the device is set to a larger number than the default value of 1 with the setCaching () method. Using default values means map

The task requests the region server as each record is processed. However, if the value is 500, then once

500 pieces of data can be sent to the client for processing, of course, the data is also determined according to your situation.

This is at the line level and is explained on our page 119.

3: limit the scanning range.

This is easy to understand, for example, we have to deal with a large number of lines (especially as an input source for mapreduce), where

When we use scan, we have the method of Scan.addFamily ();. At this time, if we just need to go to

Several columns in this column family, then we must be precise. Because too many columns can lead to a loss of efficiency.

4: close resultScanner

Of course, this will not improve our efficiency, but if it is not closed, it will affect our efficiency.

5: usage of block cache

First of all, our block cache is started by Scan.setCacheBolcks ();, lines that are accessed frequently

We should use cache blocks, but the mapreduce job scans a large number of lines, so we shouldn't use this

Yes. This block cache is different from the block I mentioned in section 4.

6: optimize the way to obtain health

Of course, the premise of using this is that we only need the row keys in the table to use it. So how to use it on page 411 is explained.

7: close WAL on Put

That's what the book says, but I personally think it's better not to use this function, because we turned it off.

The server will not write put to WAL, but directly to memstore, so that if the server fails

Our data is lost.

Thank you for your reading, the above is the content of "the process of writing and saving data in Hbase". After the study of this article, I believe you have a deeper understanding of the process of writing and saving data in Hbase, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.