Shulou(Shulou.com)06/01 Report--

This article is about how to patrol HBase. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. Cluster inspection

HBase is a NoSQL database that uses HDFS as its underlying storage. It provides database services that meet real-time and random read-write functions.

Patrol the HBase service every morning and evening to check the HMaster and RegionServer status of each cluster and whether there is a backlog of transactions.

1.1 View Requests Per Second and Num.Regions

If it is 0 as shown in the figure, you need to log in to the host to view it, which usually happens after the node host is restarted.

1.2 View standby HMaster

Each library normally has three primary nodes, one running and two standby, as shown in the figure.

1.3 View Software Attributes

As shown in the figure, there are two marked parts that are more important:

1: represents the zookeeper node of this library. If an exception occurs, the total number will be abnormal.

2: represents the average number of region in this database. In theory, merging operations will be carried out if it exceeds 300, but this operation is based on the needs of the business. The business side proposes that the database is stuttered, and then merge operations can be carried out.

1.4 View Dead Region Servers

Log in to the HBase UI of each library in terms of libraries. If there is a node with HBaseregionserver down in the current library, the following will be displayed on the page:

If this happens, it means that there is an abnormal node in the current library. You can try to log in to the failed node and check the cause of the failure (for example, the HBase process disappears, the host restarts accidentally, the host crashes, etc.)

two。 Parameter tuning

2.1 HBase HRegion maximum Compression

Hbase.hregion.majorcompaction: all HStore-

Files "maximize" the time between compression, to disable automatic maximum compression, set this value to 0.

2.2 RegionServer small Compression Thread count

Hbase.regionserver.thread.compaction.small:

The number of threads in the thread pool when regionserver Minor Compaction can be set to 5

2.3 HBase Region segmentation restrictions

Hbase.regionserver.regionSplitLimit: control the maximum number of region. If you exceed it, you cannot perform split operation. The default is 2147483647. Set 1 to disable automatic split, which can be executed manually or by scripting when the cluster is idle.

2.4 maximum HBase file size

Hbase.hregion.max.filesize: default is 10 gigabytes. If the StoreFile in any column familiy exceeds this value, the Region will be split into two, because the region split will have a short region offline time (usually less than 5 seconds). To reduce the impact on the business, it is recommended to manually time the split, which can be set larger.

2.5 HBase client write buffer

Hbase.client.write.buffer: when the client writes buffer and sets autoFlush to false, flush defaults to 2m when the client is full of buffer, and the write cache size is recommended to be set to 5m in bytes. Of course, the larger the client, the more memory it takes up.

2.6 HBase Region Server processor count

Hbase.regionserver.handler.count: this setting determines the number of threads that process RPC. The default value is 30, which can usually be increased, but not as large as possible. Too much memory will be consumed by oversetting, resulting in frequent gc or oom.

2.7 HFile block cache size

Hfile.block.cache.size: default value 0.25regionserr-

The memory size limit of ver's block cache can be appropriately increased in read-biased business. It is important to note that the sum of the value of hbase.regionserver.global.memstore.upperLimit and hfile.block.cache.size must be less than 0.8.

2.8Max size of all Memstore in RegionServer

Hbase.regionserver.global.memstore.upperLimit,hbase.regionserver.global.memstore.size: the default value is 0.4. the purpose of this parameter is to prevent excessive memory footprint. When the total memory occupied by all region memstores in ReigonServer reaches 40% of heap, HBase will force block all updates and flush these region to release the memory occupied by all memstore.

2.9 Memstore refreshed low water mark

Hbase.regionserver.global.memstore.lowerLimit,hbase.regionserver.global.memstore.size.lower.limit: the default value of 0.35 is the same as upperLimit, except that lowerLimit does not flush all Heap when the memory occupied by memstores of all region reaches 35% of Heap. It will find a region that takes up the largest amount of memstore memory and do individual flush. At this time, writing updates will still be block. LowerLimit is a remedy before all region forces flush to cause performance degradation. In the log, it appears as "* * Flush thread woke up with memory above low water"

2.10 HBase Memstore refresh size

Hbase.hregion.memstore.flush.size: such as memst-

If the ore size exceeds this value (in bytes), the Memstore will be flushed to disk. The purpose of this parameter is to flush all memstore of a Region when the sum of all the memstore sizes in that region exceeds the specified value. RegionServer's flush processes requests asynchronously by adding a queue to simulate production and consumption patterns. Then there is a problem, when the queue is too late to consume, resulting in a large backlog of requests, may lead to a sharp increase in memory, the worst case is to trigger OOM.

Thank you for reading! This is the end of the article on "how to check HBase". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.