Shulou(Shulou.com)06/03 Report--

Foreword:

It is a file system that allows files to be shared on multiple hosts over the network, allowing multiple users on multiple machines to share files and storage space.

HDFS writing process: 1. Initialize FileSystem, the client calls create () to create the file 2.FileSystem invokes the metadata node with RPC, and creates a new file in the namespace of the file system. The metadata node first determines that the file does not exist and that the client has permission to create the file, and then create a new file. 3.FileSystem returns DFSOutputStream, the client writes data, and the client starts writing data. 4.DFSOutputStream divides the data into blocks and writes it to data queue. Data queue is read by Data Streamer and tells the metadata node to allocate data nodes to store data blocks (3 blocks are replicated by default). The assigned data nodes are placed in a pipeline. Data Streamer writes the block to the first data node in the pipeline. The first data node sends the data block to the second data node. The second data node sends the data to the third data node. 5.DFSOutputStream saves the ack queue for the outgoing data block, waiting for the data node in the pipeline to tell you that the data has been written successfully. 6. When the client finishes writing data, the close function of stream is called. This operation writes all data blocks to the data node in pipeline and waits for ack queue to return success. Finally, the metadata node is notified that the write is complete. 7. If the data node fails in the process of writing, close the pipeline, put the data block in the ack queue into the beginning of the data queue, and the current data block is given a new mark by the metadata node in the already written data node, the error node can detect that its data block is outdated and will be deleted after restart. Failed data nodes are removed from the pipeline, and additional data blocks are written to the other two data nodes in the pipeline. The metadata node is informed that the block is not replicated enough and a third backup will be created in the future.

HDFS reading process: 1. Initialize FileSystem, and then the client (client) uses the open () function of FileSystem to open the file 2.FileSystem calls the metadata node with RPC to get the data block information of the file. For each data block, the metadata node returns the address of the data node where the data block is saved. 3.FileSystem returns FSDataInputStream to the client to read the data, and the client calls the read () function of stream to start reading the data. The 4.DFSInputStream connection holds the nearest data node of the first data block of this file, and data reads from the data node to the client (client) 5. When this data block is read, DFSInputStream closes the connection to this data node and then connects to the nearest data node of the next data block in this file. 6. When the client finishes reading the data, it calls the close function of FSDataInputStream. 7. In the process of reading data, if the client has an error communicating with the data node, it attempts to connect to the next data node that contains this data block.

Reference:

Https://blog.csdn.net/gaijianwei/article/details/45918337

Https://www.cnblogs.com/xubiao/p/5579080.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.