Shulou(Shulou.com)06/01 Report--

How to solve the HBase RIT problem in production in HBase operation and maintenance. For this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem find a simpler and easier way.

1. Phenomenon:

Last night, the cluster crashed, and there was not enough memory, causing HBase RegionServer to hang!

The CDH HBase Master(active) node is red and displays the message:HBase Regions In Transition Over Threshold.

Then I knew I had met HBase RIT again.

2. Common solutions:

2.1 Restart HBase, after trying twice, find that HBase Master(active) node is still red

Although we can connect to HBase, the query speed (dbeaver tool +Phoenix) is not very slow,

Cache of region boundaries are out of date.

2.2 At the master node: the user who uses the hbase process

su - hbase

hbase hbck -fixAssignments

Used to fix region assignments errors, observe more and more RIT Regions found, terminate commands in time

Also tried hbase hbck, hbase hbck -repair, log brush for a long time also did not solve.

2.3 Check the master node logs and find the following logs

2018-08-21 09:50:47,924 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fhadoop49%2C60020%2C1534734073978-splitting%2Fhadoop49%252C60020%252C1534734073978.null0.1534762936638=last_update = 1534816154977 last_version = 22 cur_worker_name = hadoop47,60020,1534815723497 status = in_progress incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0}

Because the log brush quickly, plus is info level, just started really did not pay attention!

Later, by analyzing the HBase master web interface, we found that:

a.RIT regions are all on hadoop49 machines

b. The master log also shows the splitting log of the hadoop49 machine.

Always in the in_progress state

(Observe for about 10 minutes, keep brushing info level and this state)

3. Violent resolution:

3.1 Directly use hdfs command to find logs first, then rm delete (remove to recycle bin)

hadoop36:hdfs:/var/lib/hadoop-hdfs:>hdfs dfs -ls hdfs://nameservice1/hbase/WALs/*splitting

Found 1 items

-rw-r--r-- 3 hbase hbase 21132987 2018-08-20 19:02 hdfs://nameservice1/hbase/WALs/hadoop49,60020,1534734073978-splitting/hadoop49%2C60020%2C1534734073978.null0.1534762936638

hadoop36:hdfs:/var/lib/hadoop-hdfs:>

hadoop36:hdfs:/var/lib/hadoop-hdfs:>

hadoop36:hdfs:/var/lib/hadoop-hdfs:>hdfs dfs -rm hdfs://nameservice1/hbase/WALs/hadoop49,60020,1534734073978-splitting/hadoop49%2C60020%2C1534734073978.null0.1534762936638

18/08/21 12:46:15 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/hbase/WALs/hadoop49,60020,1534734073978-splitting/hadoop49%2C60020%2C1534734073978.null0.1534762936638' to trash at: hdfs://nameservice1/user/hdfs/.Trash/Current/hbase/WALs/hadoop49,60020,1534734073978-splitting/hadoop49%2C60020%2C1534734073978.null0.1534762936638

hadoop36:hdfs:/var/lib/hadoop-hdfs:>

3.2 Restart HBase, wait for a while, everything is normal, and ensure that HBase provides external services. 3.3 Because we delete the HLOG file, it will inevitably lose data, so use MCP real-time middleware, web interface custom data re-brush job(last night 19:00~21:00 fault range time), restore data.

About how to solve HBase RIT problems in production in HBase operation and maintenance is shared here. I hope the above content can help you to some extent. If you still have a lot of doubts, you can pay attention to the industry information channel for more relevant knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.