Shulou(Shulou.com)06/03 Report--

1. The client or user opens the file that needs to be read by calling the Open () method of the FileSystem object, which is an object obtained by the HDSF distributed system.

2.FileSystem calls NameNode through the remote protocol to determine the location of the first few Block of the file. For each block,NameNode, a metadata containing Block is returned, and then the DataNode is sorted according to the distance (offSet offset) defined above. If the Client itself is a DataNode, then the data will be read first from the local DataNode node. After completing the above work, HDFS returns a FSDatalnputStream to the client to read from the FSDatalnputStream. FSDatalnputStream wraps a class DFSInputStream that manages the DataNode and NameNode I _ Unio operations.

3.NameNode returns an address containing data information to the client, and the client creates an FSDDatalnputStream object based on the address to start reading the data.

According to the DataNode address of the first few Block, 4.FSDatalnputStream connects to the recent DataNode and starts to read the data from scratch. The client will repeatedly call the read () method to read the data in DataNode in the form of stream.

5. When reading the end of Block, FSDatalnputStream closes the connection to the current DataNode and finds out whether the DataNode location information of the next Block can be read. This gives the client the impression that the data is read continuously, that is, when reading, it starts to look up the address of the next block. When reading is complete, you need to use the close () method to close FSDatalnputStream.

HDFS writing process:

1.Client requests to create a file by calling the Create () method of the FileSystem class

2.FileSystem creates a new file in NameNode by issuing a remote request to NameNode, but no blocks are associated at this time. NameNode does a lot of checks to ensure that the file to be created is not already in the system, and to check whether there are considerable permissions to create the file. If all these are checked, NamdNode will record the information of the new file. FileSystem returns a FSDatalnputStream to the client to write data, and FSDatalnputStream will wrap a DFsOutputStream for communication between DataNode and NameNode. Once the file creation fails, it will return a l0Exception to the client indicating that the file creation failed and stop the task.

3. The client starts to write data FSDatalnputStream divides the data to be written into packets and writes it to the intermediate queue, in which the data is read by DataStramer. The responsibility of DataStreamer is to let NameNode allocate a new block to store the data by looking up the corresponding DataNode. The stored data will be backed up and stored in other nodes to complete data writing.

4.FSDatalnputStream maintains an internal queue about packets, which stores information waiting to be confirmed by DataNode. Once the data information is successfully written to DataNode, it will be removed from the queue.

5. After the data has been written, the customer will call the close method, and write the residual data together through a method flush before completing the write through NameNode. And NameNode determine the location information written and return the success status.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.