Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to repair the damage to the edits file in hadoop. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Some time ago, the company's hadoop cluster went down and found that the namenode disk was full. After clearing part of the space, the restart fails when the cluster is restarted.

It is also found that the cluster Secondary namenode service is also broken, causing all operations log to continue to write to edits.new files, and when the cluster goes down, the file size has reached a crazy 70GB.. Restart cluster error failed to load edits file. The reason for analyzing the error in the load file is that insufficient disk leads to the downtime of the last written log which is only half written. Due to the incomplete log, hadoop reported an error reading the edits file when it started loading the file again. Because the edits.new file is too large and stores a lot of operational log, it must be repaired.

An attempt was made to delete the last few lines of the file, but an error was reported. So look at the source code to analyze the edits file structure and find that it is in binary format, the first behavior version number, and then the log record content during the operation of hadoop, which consists of opcode + length (optional) + other items.

Edits file format analysis chart

Solution.

The method to report the error location in the source code is to read the last location of the file in the org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits (EditLogInputStream edits) method because of the lack of part of the data to report an error, so take out this part of the code separately, remove the business operation part, leave only the reading process, record the file length len before the exception, and then copy the contents from 0 to len into a new edits file. Start the hadoop cluster, successful!

NameNode initiates loading metadata scenario analysis

Call FSNamesystemm in the NameNode function to read dfs.namenode.name.dir and dfs.namenode.edits.dir to build FSDirectory.

FSImage classes recoverTransitionRead and saveNameSpace implement metadata inspection, loading, memory merging and persistent storage of metadata, respectively.

SaveNameSpace writes metadata to disk. The steps are as follows: first, rename the current directory to lastcheckpoint.tmp;, then create a new current directory and save the file; finally, rename lastcheckpoint.tmp to privios.checkpoint.

The process of checkPoint: Secondary NameNode tells nameNode to generate an edit log file, edits.new, after which all log operations are written to the edits.new file. Next, Secondary NameNode will download the fsimage and edits files from namenode, merge them to produce a new fsimage.ckpt;, and then Secondary will upload the fsimage.ckpt file to namenode. Finally, namenode will rename fsimage.ckpt to fsimage,edtis.new to edits

PS:

The latest CDH version of hadoop cluster startup allows you to recover edits files and skip error log

This is the end of the article on "how to repair the damage of edits files in hadoop". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.