Shulou(Shulou.com)06/01 Report--

This article focuses on "hadoop hbase cluster power outage data blocks destroyed can not start how to solve", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "hadoop hbase cluster power outage data block is destroyed can not start how to solve it"!

The cluster machine is unexpectedly powered off and restarted, which causes hbase not to start normally and throws a reflect invocation exception, which may be interrupted in the middle of the operations such as insert or merge, resulting in incomplete format of some data files or incomplete block blocks on the hdfs.

Checked the relevant information on the Internet, suspected that it may be closed before some unsubmitted changes stored in the log file data written half of the file is incomplete, so change hbase.hlog.split.skip.errors to true to try.

An explanation of the role of this parameter:

When the server crashes and restarts, there will be a playback process, which replays all the log recorded under / hbase/WAL/ and merges them into each region. If error occurs during playback and this parameter is false, then exception will output to the outer layer and playback fails.

Unfortunately, the hbase cluster still fails to start properly after changing this parameter.

Then consider other reasons, first look at the 60010 monitoring page when hbase starts.

Some region FAILED_OPEN errors were found. There were 200 region in the its007-meta table, and only 199 were started successfully.

It seems to think of something, by the way, it is likely that the data file format of this region is incorrect, so first check whether its file on hdfs is normal.

Sure enough, if you look at the 50070 page of hadoop, you will notice that there are two blocks of data error on the specific path of the hadoop file system.

(link to articles about hbase's directory on hdfs: HBase's directory tree on HDFS)

Solution:

1. Run hadoop fsck /-files to check the hdfs file

two。 A file corruption was found in the / hbase/oldWALs directory

Run hadoop fsck /-delete to clean up corrupted files

3. Run hbase hbck-details to view the hbase profile and find that Table its007-meta has a region load failure

4. Run hbase hbck-fixMeta to try to repair the system metadata table

5. Run hbase hbck-fix to attempt to fix region data inconsistencies.

6. Run hbase hbck-details again and find that the problem is still not fixed and the region still fails to load.

Therefore, the files with errors under the region are removed directly and temporarily moved to the hdfs root directory.

Hadoop fs-move / hbase/data/default/its007-meta/fe6463cba743a87e99f9d8577276bada/meta/9a853fdbe13046fca194051cb9f69f9b /

Fe6463cba743a87e99f9d8577276bada is the name of region.

9a853fdbe13046fca194051cb9f69f9b is an error HFile under region, with a size of 800k. (note: there can be multiple HFile under a region)

7. Run hbase hbck-fix to reload the previously failed region, which completes the repair and discards the faulty HFile

At this point, I believe that everyone on the "hadoop hbase cluster power outage data blocks destroyed can not start how to solve", might as well to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.