Shulou(Shulou.com)06/01 Report--

What is the working mechanism of namenode and secondarynamenode in Hadoop? in order to solve this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

1) proc

2) FSImage and Edits

Nodenode is the brain of HDFS. It maintains the directory tree of the entire file system, as well as all the files and directories in the directory tree. This information is stored in the file system in two kinds of files: one is the namespace image (also known as file system image, File System Image,FSImage), that is, the complete snapshot of HDFS metadata. Every time NameNode starts, the latest namespace image is loaded by default. The other is the editing log (Edit log) of the command space image.

The FSImage file is actually a permanent checkpoint of the file system metadata, but not every write operation updates the file, because FSImage is a large file, and frequent writes can make the system extremely slow. The solution is that NameNode prewrites only the changes, that is, to the edit log of the namespace image. As time goes by, editing logs will become larger and larger, and in the event of a failure, it will take a lot of time to roll back and forth, so just like traditional relational databases, you need to merge FSImage and edit logs on a regular basis. If NameNode does the merge operation, then NameNode may not be able to provide sufficient resources when providing services to the cluster. In order to completely solve this problem, SecondaryNameNode promised.

3) the first stage: namenode startup

(1) after starting namenode formatting for the first time, create fsimage and edits files. If it is not the first time to start, load the editing log and image file directly into memory.

(2) the request of the client to add, delete and modify the metadata.

(3) namenode records the operation log and updates the rolling log.

(4) namenode adds, deletes, modifies and queries the data in memory.

4) the second stage: Secondary NameNode work

(1) Secondary NameNode asks if namenode needs checkpoint. Bring back namenode directly to see if the result is checked.

(2) Secondary NameNode requests to execute checkpoint.

(3) Secondary NameNode guides namenode to scroll update and edit the edits log that is being written.

(4) Secondary NameNode loads the FSImage file, plays back the editing log, merges it into FSImage, and writes the new FSImage file to disk after compression.

(5) copy fsimage to namenode.

(6) namenode renamed fsimage to fsimage.

By default, this process occurs every hour, or when NameNode's edit log file arrives at the default 64MB.

The answer to the question about the working mechanism of namenode and secondarynamenode in Hadoop is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.