Shulou(Shulou.com)05/31 Report--

This article mainly explains the "Hadoop upload and download efficiency optimization method is what", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "what is the upload and download efficiency optimization method of Hadoop"!

I. Overview

The primary technical problem faced by the cloud disk system based on any platform is the optimization of client upload and download efficiency. The cloud disk system based on Hadoop is affected by the read-write mechanism of Hadoop files. API provided by Hadoop is used to access the HDFS file system. The default is to read files sequentially and block by default, and to write sequentially when writing.

Second, the reading and writing mechanism

First of all, let's look at the file reading mechanism: although DataNode implements the horizontal expansion of file storage space and multi-copy mechanism, the default API interface of Hadoop does not provide multi-DataNode parallel reading mechanism for reading a single specific file. Cloud disk clients based on the API interface provided by Hadoop naturally face the same problem. The file reading process of Hadoop is shown in the following figure:

Use the client development library provided by HDFS to initiate a RPC request to the remote Namenode

Namenode returns part or all of the block list of the file as appropriate, and for each block,Namenode, it returns the datanode address with a copy of the block.

The client development library selects the datanode closest to the client to read the block.

After reading the data of the current block, close the connection with the current datanode and find the best datanode for reading the next block

When the block of the list is read and the file reading is not finished, the client development library continues to get the next batch of block lists from Namenode.

After reading a block, checksum verification is performed, and if there is an error in reading the datanode, the client notifies the Namenode and then continues reading from the next datanode that owns the copy of the block.

The key point to note here is that multiple Datanode reads sequentially.

Secondly, let's look at the writing mechanism of the file:

Use the client development library provided by HDFS to initiate a RPC request to the remote Namenode

Namenode will check whether the file to be created already exists and whether the creator has permission to operate. If it succeeds, it will create a record for the file, otherwise it will cause the client to throw an exception.

When the client starts to write to the file, the development library splits the file into multiple packets, manages the packets internally as "data queue", and requests a new blocks from the Namenode to get the appropriate datanodes list to store the replicas, depending on the size of the replication set in the Namenode.

Start writing packet to all replicas in the form of pipeline (pipes). The development library writes the packet to the first datanode as a stream, which stores the packet and then passes it to the next datanode in the pipeline until the last datanode, which writes data in pipelined form.

After the last datanode is successfully stored, an ack packet is returned, which is passed to the client in pipeline, and the "ack queue" is maintained inside the client's development library. When the ack packet returned by datanode is successfully received, the corresponding packet is removed from "ack queue".

If a datanode fails during transmission, the current pipeline will be closed, the failed datanode will be removed from the current pipeline, the remaining block will continue to be transmitted in the form of pipeline in the remaining datanode, and the Namenode will allocate a new datanode to maintain the number set by replicas.

Keywords: the development library writes the packet to the first datanode as a stream, and the datanode passes it to the next datanode in the pipeline, knowing the last Datanode. This way of writing data is pipelined.

III. Solutions

1. Download efficiency optimization

Through the analysis of the above read-write mechanism, we can find that the optimization of download efficiency of cloud disk client segment based on Hadoop can start from two levels:

1. The overall level of files: parallel access to multi-thread (multi-process) and multi-file parallel reading.

2.Block block read: rewrite Hadoop interface extension, multi-Block parallel read.

two。 Upload efficiency optimization

Upload efficiency optimization can only use parallel processing at the overall level of files, and does not support multi-Block parallel reading with sub-Block mechanism.

Thank you for your reading, the above is "what is the upload and download efficiency optimization method of Hadoop", after the study of this article, I believe you have a deeper understanding of what the upload and download efficiency optimization method of Hadoop is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.