Shulou(Shulou.com)05/31 Report--

This article shares with you the content of a sample analysis of hbase performance optimization. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Query cache

The default value of the caching attribute for Scan is 1, which means that the scanner fetches one record at a time from the region server to match. We can set caching to a much larger value than 1. For example, if you set it to 500, you can crawl 500 at a time, and it is important to note that the higher the value, the more memory the server will spend.

HTableInterface hTable=getHTable (tableName)

Scan scan=new Scan ()

/ * set cache * /

Scan.setCaching (StaticConfig.getIControl_hbase_cache ())

ResultScanner scanner= hTable.getScanner (scan)

Multithreaded configuration

Hbase.regionser.handler.count

The number of RPC listener instances in the RegionServer. For master, this property is the number of processing threads (handler) offered to the master. The default value is 10.

According to the business scenario of the regulation layer, one matching query of freight rates will produce 4 hbase concurrent queries. If there are 20, there may be 80 concurrency, which is equivalent. In addition to increasing this parameter appropriately, it also has a direct relationship with the number of clusters and the configuration of servers. It is estimated that the more clusters, the higher the number of server CPU cores, and the stronger the concurrent processing capacity.

Pre-partition

HRegion is the smallest unit of distributed storage and load balancing in Hbase. The smallest unit means that different Hregion can be distributed on different HRegion server. However, a Hregion will not be split into multiple server.

Hbase.hregion.max.filesize

The maximum value of HstoreFile. If the storage file of any column family in Region exceeds this limit, it will be split into two region. The default is 268435456 (256x1024x1024), which is 256m.

Our regulatory files are relatively small, and more regulatory files are needed to reach the maximum partition limit of 256m. In order to improve the concurrency, we need to generate multiple hregion to save and process data without reaching the partition limit. Here we use the pre-partitioning feature of hbase.

Example:

Configuration conf = HBaseConfiguration.create ()

HBaseAdmin admin = new HBaseAdmin (conf)

HTableDescriptor desc = new HTableDescriptor (

Bytes.toBytes (tablename))

HColumnDescriptor coldef = new HColumnDescriptor (

Bytes.toBytes (colfamily))

Admin.createTable (desc, Bytes.toBytes (1L), Bytes.toBytes (10L), 10)

/ / divide the region according to the first character

Desc.setValue (HTableDescriptor.SPLIT_POLICY

KeyPrefixRegionSplitPolicy.class.getName ()

Desc.setValue ("prefix_split_key_policy.prefix_length", "1")

Thank you for reading! This is the end of this article on "sample Analysis of hbase performance Optimization". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.