Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about how HBase solves the problem that the Region Server Compact process takes up a lot of network exit bandwidth. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

After HBase version 0.92, the Compact process of RegionServer is divided into smallcompaction and large compaction according to the file size to be merged, which may cause Compact to occupy too much network egress bandwidth when the write volume of the cluster is large. This article will describe in detail the troubleshooting process and solutions to this problem encountered in the use of the cluster.

1. Find a problem

During the operation of the HBase cluster (version 0.94.0), it is found that the network egress bandwidth of the five Region Server is often maintained above 100MB/s, close to the limit of the network card; at the same time, the load load of the Region Server machine is also very high, reaching 3050 at peak time.

two。 Troubleshoot the problem

1. During the actual operation of the cluster, it is observed that the average write traffic of each network card on the RegionServer server is about 60MB/s (at this time, the write volume is already very large); the read traffic 90MB/s, and sometimes even break through 100MB/s (Note: each machine is a gigabit network card)

2. It is observed that the actual amount of data written is about 5w tps per second, and the average size of a single record is 1KB, which will probably occupy the bandwidth request of the network card entrance about 50MB/s, which is consistent with the observed phenomenon.

3. It is observed that the query volume is about 6w qps per second, and the average size of a single record is 1KB, which will probably occupy the bandwidth request volume of the network card exit about 60MB/s. It is strange that it is actually observed that there are network exit bandwidth requests that are close to or even exceed 100MB/s, which is more than the network exit bandwidth of about 40MB/s.

4. After analysis and investigation, it is determined that the reason for the above process may be that the HBase server frequently triggers the compaction process due to excessive writes, and compaction needs to read HBase data, so it occupies a considerable part of the network egress bandwidth.

5. Combined with the analysis of the relevant source code org/apache/hadoop/hbase/regionserver/CompactSplitThread.java, we decided to make changes to the HBase cluster configuration (see the next section for details). The main purpose is to reduce the occurrence of compaction.

6. Next, it is observed that the network utilization of RegionServer decreases obviously, and the general inlet and outlet bandwidth can be maintained below 70MB/s.

3. Solve the problem

After version 0.92 of HBase, a configuration option for compact has been added. Compact is divided into two thread pools: smallcompaction and large compaction. (by default, each has a thread. For more information on the source code, please see org/apache/hadoop/hbase/regionserver / CompactSplitThread.java). Because the compact process needs to read data from the HBase cluster, the actual operation results in a large amount of network egress traffic occupied by compact. The solution is to selectively shut down smallcompaction or large compaction. There are two changes that can be resolved:

1) option 1

(1) modify hbase.regionserver.thread.compaction.throttle to a large value (such as 50GB) to force all compact to become small compaction, thus reducing the pressure on compact.

(2) set the number of smallcompaction and large compaction threads to 1 to reduce the pressure on compact (can not be configured, the system will initialize it to 1 by default).

Procedure:

To prepare the hbase-site.xml file, add or modify the following options:

Hbase.regionserver.thread.compaction.throttle

53687091200

Hbase.regionserver.thread.compaction.small

one

Hbase.regionserver.thread.compaction.large

one

Restart the cluster for the configuration to take effect.

2) option II

Set the number of small compaction threads to 0, thus turning off small compaction, leaving only large compaction, which can also reduce the pressure on compact.

Procedure:

To prepare the hbase-site.xml file, add or modify the following options:

Hbase.regionserver.thread.compaction.small

0

Restart the cluster for the configuration to take effect.

This is how the HBase shared by the editor solves the problem that the Region Server Compact process takes up a lot of network egress bandwidth. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.