Shulou(Shulou.com)06/01 Report--

This article will explain in detail about the persistence mechanism of Hadoop Namenode metadata and the role of SecondaryNamenode. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

We all know that namenode is used to store metadata, it is not used to store real data.

So how does his metadata persist!

FsImage

The image file of the file system is called fsImage, which includes the mapping of file and block information, as well as the attribute information of the file system.

During the startup of datanode, block information is first registered with namenode, which is stored in the FsImage of namenode.

EditLog

Every change to the file system, such as adding files, deleting files, and so on, is written to the Edit Log problem.

Similarly, modifying the copy factor configuration of the file system will also be written to the Edit Log file.

Both EditLog and fsImage files are stored in the local path of the file system.

Namenode stores image and block mapping information for the entire file system in memory.

Metadata can be merged, so namenode has the memory of 4GB, which is enough to store a large number of files and directories.

What happens during namenode startup?

1. Read FsImage and EditLog files from the hard disk.

2. Write all the operation information in the EditLog file to FsImage. Generate a new FsImage file-this operation is called checkpoint

3. Generate a new empty EditLog file

Checkponit when namenode starts

The interaction between fsImage and EditLog during the operation of namenode

SecondaryNamenode many people understand that the function of SecondaryNamenode is to back up namenode. When namenode goes down, it can be quickly recovered with secondarynamenode. In fact, the important role of SecondaryNamenode is to merge FsImage and EditLog files on a regular basis.

Consider a scenario. When our namenode goes down. We need to read the EditLog file and merge the fsImage file for recovery. If the EditLog file is very large, then it may take a long time to read the EditLog file, so that our failure recovery is very poor. So SecondaryNameNode, regularly merge FsImage and EditLog files, and replace the old FsImage file on namenode, generate a new EditLog file, replace the old EditLog file. This ensures that the files on SecondaryNameNode are up-to-date. When there is an outage, it can be recovered quickly.

On the Hadoop Namenode metadata persistence mechanism and what the role of SecondaryNamenode is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.