Shulou(Shulou.com)05/31 Report--

This article mainly introduces how the HDFS Namenode is composed, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1.Namenode composition

The fsimage image file contains indoe information for all directories and files of the entire HDFS file system. For files, it includes block description information, modification time, access time, etc.; for directories, it includes modification time, access control information (directory users, groups, etc.) and so on.

Edit file is mainly to record all kinds of update operations of HDFS when NameNode has been started, and all write operations performed by HDFS client will be recorded in edit file.

2. 1.X SecondaryNamenode

Secondarynamenode tells namenode to generate a new log file (edits.new), and all subsequent logs are written to the new log file.

Secondarynamenode uses http get to get the fsimage file and the old log file (edits) from the metadata node.

Secondarynamenode loads the fsimage file into memory, performs the operation in the log file (edits), and then generates a new fsimage file.

Secondarynamenode sends the new fsimage file back to namenode by http post.

Namenode can replace the old fsimage file and the old log file (edits) with the new fsimage file and the new log file (generated in the first step) (edits.new-> edits), and then update the fstime file to write the time of the checkpoint.

In this way, the fsimage file in the metadata node stores the latest checkpoint metadata information, and the log file starts over and will not become very large.

The above steps can effectively reduce the size of the edits file, thus reducing the time it takes to start the execution of merge.

Note: http://p-x1984.iteye.com/blog/860290

3.2.X HA and merge (NFS and JQM)

NFS,shared NN EDITS LOG

Thank you for reading this article carefully. I hope the article "how HDFS Namenode is made up" shared by the editor will be helpful to everyone. At the same time, I hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.