Shulou(Shulou.com)06/01 Report--

This article mainly explains the working mechanism of NameNode and the working mechanism of DataNode copy. The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and study the working mechanism of NameNode and the working mechanism of DataNode copy.

How NameNode (hereinafter referred to as NN) works:

1. The operation of uploading data is saved in memory cache and written to edits.log file at the same time

2. The edits.log file is a temporary log file, and when the edits.log file reaches a certain size, the data will be written to another edits.log2 file, so multiple small edits.log files will be generated.

3. Multiple small edits.log files are finally saved in the local fsimage file through the SecondaryNameNode (hereinafter referred to as SN) node

4. NN sends checkpoint requests to SN at regular intervals (the combination of fsimage and edits.log)

5. SN downloads the fsimage and edits.log files from NN, and then requests the edits.log file to change the file name to edits.new

6. SN merges fsimage and edits.log in memory to generate a new fsimage.checkpoint and notify NN

7. NN downloads fsimage.checkpoint from SN after receiving the notification

8. NN changes the fsimage.checkpoint and edits.new files back to their original names

Copy generation mechanism of DataNode (hereinafter referred to as DN):

DN uses pipeline (pipeline) mechanism to copy the data. The client has only one copy when it is submitted to DN. DN forms a pipeline according to each host from NN. Once data is transferred to the first node of DN, DN will copy data asynchronously to other DN nodes in the pipeline. Only when all nodes have completed the copy, the pipeline will be considered successful. Otherwise, DN will notify NN of replica failure, and NN will receive a DN failure request and make an adjustment according to the copied successful node and failed node to re-form a new pipeline (for example, there are 3 nodes in transmission, and if 3 nodes fail, the pipe fails. When reselecting, NN will manage the replica of the second node that has been successfully transferred and other nodes that are not the original third node.

Thank you for your reading, the above is the content of "the working mechanism of NameNode and the working mechanism of DataNode copy". After the study of this article, I believe you have a deeper understanding of the working mechanism of NameNode and the working mechanism of DataNode copy, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.