Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the official recommendations of hbase". In daily operation, I believe many people have doubts about the official recommendations of hbase. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the questions of "what are the official recommendations of hbase?" Next, please follow the editor to study!

1. The number of region on regionServer: the official recommendation is 20-200 region, but the following is the following

How to roughly calculate region on regionServer: there is a formula

HBase 0.98.x

((RS Xmx) * hbase.regionserver.global.memstore.size) / (hbase.hregion.memstore.flush.size * (# column families)) +

HBase 0.94.x

((RS Xmx) * hbase.regionserver.global.memstore.upperLimit) / (hbase.hregion.memstore.flush.size * (# column families)) +

It seems that the default values of both hbase.regionserver.global.memstore.size and hbase.regionserver.global.memstore.upperLimit are 0.4. that is to say, if the memory allocation of my RS is 16GB and the rest is the default value, there is only one column family, and there are 51 region in version 0.98. why is it calculated in this way? In fact, this is an extreme case. In fact, the function of this parameter hbase.regionserver.global.memstore.upperLimit is to prevent excessive memory consumption. When the total memory occupied by all region memstores in ReigonServer reaches 40% of heap, HBase will force block all updates and flush these region to release the memory occupied by all memstore. This formula assumes that memstore reaches the maximum value, in order not to happen above. It is the same as upperLimit, except that lowerLimit does not flush all memstore when the memstores of all region reaches 35% of the memory of Heap. It will find a region that takes up the largest amount of memstore memory and do individual flush. At this time, writing updates will still be block. LowerLimit is a remedy before all region forces flush to cause performance degradation. In the log, it appears as "* * Flush thread woke up with memory above low water."

Parameter adjustment will affect read and write. If the write pressure often exceeds this threshold, turn down the read cache hfile.block.cache.size to increase the threshold, or do not modify the read cache size when there is a large margin of Heap.

If the threshold is not exceeded in the case of high pressure, it is recommended that you properly reduce this threshold and then do the pressure test to ensure that the number of triggers is not too much, and then increase the hfile.block.cache.size to improve the reading performance when there is more Heap margin.

Parameter adjustment will affect read and write. (from experience below) if the write pressure often exceeds this threshold, reduce the read cache hfile.block.cache.size, increase the threshold, or do not modify the read cache size when there is a large margin of Heap.

If the threshold is not exceeded in the case of high pressure, it is recommended that you properly reduce this threshold and then do the pressure test to ensure that the number of triggers is not too much, and then increase the hfile.block.cache.size to improve the reading performance when there is more Heap margin.

Hfile.block.cache.size represents the percentage of the size of the Heap occupied by storefile's read cache, and 0.2 represents 20%. This value directly affects the performance of data reading. Hbase.regionserver.global.memstore.upperLimit and this parameter affect one write and the other read. If the two values add up to more than 80-90%, there will be a risk of OOM.

2 、 Maximum region size

The official recommendation is maximum region size is 10-20 GB 5-10Gb is optimal.

After version 0.98, some optimization mechanisms have been added to make the region size larger.

At this point, the study of "what are the official recommendations of hbase" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.