Shulou(Shulou.com)06/01 Report--

This article introduces how to understand HDFS in Hadoop. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

First, the basic concept of HDFS 1.1, data block (block)

The default most basic unit of storage for HDFS (Hadoop Distributed File System) is 64m blocks of data.

Like a normal file system, files in HDFS are stored in 64m blocks.

Unlike ordinary file systems, in HDFS, if a file is less than the size of a block, it does not take up the entire block storage space.

1.2.metadata nodes (Namenode) and data nodes (datanode)

Metadata nodes are used to manage the namespace of the file system

It saves the metadata of all files and folders in a file system tree.

This information will also be saved to the following files on the hard drive: namespace image (namespace image) and modification log (edit log).

It also saves which data blocks are included in a file and which data nodes are distributed. However, this information is not stored on the hard disk, but is collected from the data node when the system is started.

A data node is the place where data is actually stored in the file system.

The client (client) or metadata information (namenode) can request from the data node to write or read the data block.

It periodically reports its stored block information to the metadata node.

Slave metadata node (secondary namenode)

The slave metadata node is not a backup node when there is a problem with the metadata node, it is responsible for different things from the metadata node.

Its main function is to periodically merge the namespace image file of the metadata node and the modification log to prevent the log file from being too large. This will be believed in the narrative below.

The merged namespace image file is also saved from the metadata node in case the metadata node fails.

1.2.1. Metadata node folder structure

The VERSION file is a java properties file that holds the version number of the HDFS.

LayoutVersion is a negative integer that holds the format version number of HDFS's persistent data structure on the hard disk.

NamespaceID is the unique identifier of the file system and is generated when the file system is first formatted.

CTime is 0 here

StorageType indicates that what is stored in this folder is the data structure of the metadata node.

NamespaceID=1232737062

CTime=0

StorageType=NAME_NODE

LayoutVersion=-18

1.2.2, file system namespace image files and modification logs

When the file system client (client) writes, it is first recorded in the modification log (edit log)

The metadata node holds the metadata information of the file system in memory. After the modification log is recorded, the metadata node modifies the data structure in memory.

The modification log is synchronized (sync) to the file system before each write is successful.

The fsimage file, that is, the namespace image file, is the checkpoint of the metadata in memory on the hard disk. It is a serialized format and cannot be modified directly on the hard disk.

Similar to the mechanism of data, when the metadata node fails, the metadata information of the latest checkpoint is loaded into memory from fsimage, and then the operations in the modification log are re-performed one by one.

The slave metadata node is used to help the metadata node checkpoint the metadata information in memory to the hard disk.

The process of checkpoint is as follows:

The metadata node is notified from the metadata node to generate a new log file, and subsequent logs are written to the new log file.

From the metadata node, use http get to get the fsimage file and the old log file from the metadata node.

Load the fsimage file into memory from the metadata node, perform the operations in the log file, and then generate a new fsimage file.

Return the metadata node from the new fsimage file using http post

The metadata node can replace the old fsimage file and the old log file with the new fsimage file and the new log file (generated in the first step), and then update the fstime file to write the time of the checkpoint.

In this way, the fsimage file in the metadata node stores the latest checkpoint metadata information, and the log file starts over and will not become very large.

1.2.3. Directory structure of slave metadata nodes

1.2.4. Directory structure of data nodes

The VERSION file format of the data node is as follows:

NamespaceID=1232737062

StorageID=DS-1640411682-127.0.1.1-50010-1254997319480

CTime=0

StorageType=DATA_NODE

LayoutVersion=-18

Blk_ saves the data blocks of HDFS, in which the specific binary data is saved.

Blk_.meta saves the attribute information of the data block: version information, type information, and checksum

When the number of blocks in a directory reaches a certain number, a subfolder is created to hold the block and block attribute information.

II. Data flow 2.1.The process of reading files

The client (client) uses the open () function of FileSystem to open the file

DistributedFileSystem calls the metadata node with RPC to get the block information of the file.

For each data block, the metadata node returns the address of the data node where the data block is saved.

DistributedFileSystem returns FSDataInputStream to the client to read the data.

The client calls the read () function of stream to start reading the data.

The DFSInputStream connection holds the nearest data node of the first data block of this file.

Data reads from the data node to the client (client)

When this data block is read, DFSInputStream closes the connection to this data node and then connects to the nearest data node of the next data block in this file.

When the client finishes reading the data, it calls the close function of FSDataInputStream.

In the process of reading data, if the client has an error communicating with the data node, it attempts to connect to the next data node that contains this data block.

Failed data nodes will be logged and will not be connected later.

2.2. The process of writing a document

The client calls create () to create the file

DistributedFileSystem invokes the metadata node with RPC to create a new file in the file system's namespace.

The metadata node first determines that the file does not exist and that the client has permission to create the file, and then creates a new file.

DistributedFileSystem returns DFSOutputStream, and the client is used to write data.

The client begins to write data, and DFSOutputStream divides the data into blocks and writes it to data queue.

Data queue is read by Data Streamer and tells the metadata node to allocate data nodes to store data blocks (3 blocks are replicated by default). The assigned data nodes are placed in a pipeline.

Data Streamer writes the block to the first data node in the pipeline. The first data node sends the data block to the second data node. The second data node sends the data to the third data node.

DFSOutputStream saves the ack queue for the outgoing data block, waiting for the data node in the pipeline to tell you that the data has been written successfully.

If the data node fails during the write process:

Close pipeline and put the data blocks in ack queue into the beginning of data queue.

If the current data block is given a new mark by the metadata node in the written data node, the error node can detect that its data block is outdated and will be deleted after restart.

Failed data nodes are removed from the pipeline, and additional data blocks are written to the other two data nodes in the pipeline.

The metadata node is informed that the block is not replicated enough and a third backup will be created in the future.

When the client finishes writing data, the close function of stream is called. This operation writes all data blocks to the data node in pipeline and waits for ack queue to return success. Finally, the metadata node is notified that the write is complete.

On how to understand the HDFS in Hadoop to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.