Shulou(Shulou.com)06/03 Report--

Recently, I have continued to read "Hadoop 2.x HDFS Source Analysis" and now I have seen the third chapter, NameNode. NameNode plays the role of master in the file system of hdfs, and is responsible for many functions such as file system directory management (namespace management), block management, data node management, lease management, cache management and so on. This time, I mainly write notes about namespace management.

Basic type

The most basic class in hdfs should be INode. The final directory, concrete file, soft connection or reference after adding snapshot function are all subclasses of this class. The inheritance relationship is shown in the following figure.

Figure 1

INode: the field including parent indicates the parent INode, and also provides interfaces such as isFile (), isDirectory (), isRoot ()

INodeWithAddtionalFields: including id, name, permission, features fields to further expand the basic information

INodeFile: header and bolcks (BlockInfo [] type) are defined on the basis of the parent class. The corresponding feature can indicate whether it is under construction (underconstruction) and whether there is a snapshot (snapshot).

INodeDirectory: a childern field has been added to the parent class to indicate whether all child INode and corresponding feature can have snapshots (snapshot) and quota (quota)

INodeReference: this class and its subclasses are mainly used when the snapshot function is enabled. Before the introduction, we first introduce the snapshot. The so-called snapshot is to save the current state of this directory, so that no matter you delete, add or rename the files in this directory later, you can get all the files in the directory at this time through the snapshot.

As shown in figure 2 below

Figure 2

At t0, there is a file / A/TXT, and a snapshot snapshot:t0 corresponding to the time / A directory is established, so when the TXT is moved to the / B directory after t0, the file cannot be found directly by accessing A, but you can access the snapshot of t0 by accessing the feature array of / An INodeWithAdditionalFields object, and then you can find the corresponding location of the TXT file according to the corresponding INodeReference object.

Here WithName can be understood as the location of the file before modification, DSTReference can be understood as the location of the file after modification, they all point to a WithCount, but WithName and the former can make a many-to-one relationship, because there can be multiple snapshots of the file saved TXT, and only one DstReference represents the current path of the file (the parent object of WithCount is DstReference), and finally WithCount points to the real file / B/TXT.

About the Log

There are many details about the log, but what impresses me most is the following two points (maybe I didn't look deep enough, [cover my face]):

1) the log uses double buffering and performs logical and physical IO at the same time:

This buffer is composed of Ready (disk IO) and Current (memory IO). The thread that writes the log is responsible for writing the log to the Current. When the Current is full and needs to be saved to the hard disk, change the name of the two areas, so that the log persistence operation will not affect the current log write operation.

2) since there can be multiple places to write to the log, it is convenient to use a set journalSet that holds all the output streams when writing the log:

The journalSet object has a selectInputStreams method for selecting the log output stream that needs to be output, thus binding the journalSet to the stream that needs to be output, and then directly calling the journalSet.startLogSegment method to start writing to the log.

About FSImage

This class is responsible for saving the namespace to disk, loading the fsImage file at startup, and loading the editlog file:

1) Save the namespace:

The saved namespace saves the contents of memory to disk in the following format, as shown in figure 3

Figure 3

MagicNumber: magic number has been adopted in many technologies, equivalent to identifying what type of file this file is, such as the 'CAFEBABE' of the class file.

Sections: this is the part that actually records the data in memory, such as INode information, cache information, snapshot information, etc.

FileSummary: this section is equivalent to the metadata of sections, describing the length and starting position of each section

FileSummaryLength: this section describes the length of the FileSummary.

When NameNode starts, look back and forward, first look at the FileSummaryLength, then read the FileSummary, and then load the Section into memory according to the metadata of each section, and the MagicNumber corresponds to 'HDFSIMG1'.

2) about checkpoint

In order to prevent the namenode startup caused by loading too large editlog files to be too slow, like in the database, hdfs also has its own checkpoint mechanism (of course, there is no rollback segment in the database, so it is relatively simple).

In the case of non-HA, the checkpoint operation is performed by the SecondaryNamenode process, but in the case of HA, the checkpoint operation is carried out by standbynamenode, and then passed to active namenode after the operation is completed.

The above cases are the implementation of namenode in hadoop2.6.0, and the specific implementation will be different with different versions. The relevant screenshots have been packaged and uploaded.

2017.2.19

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.