Shulou(Shulou.com)05/31 Report--

This article mainly explains "the size of map fragments should be the same as the block size". The explanation in the article is simple and clear, easy to learn and understand. Please follow Xiaobian's train of thought to study and learn "should map fragments be the same size as blocks?"

All the time-saving optimizations are mainly about the data local optimization adopted by Hadoop to avoid wasting valuable network bandwidth, but sometimes for a Map task input, three nodes storing a backup of a HDFS block may be running other map tasks, and the job scheduling, that is, the so-called JobTracker needs to find an idle machine in the same rack to run the map task in one of the three backups.

So we should be clear about why the optimal shard size should be the same as the block size: because this mechanism ensures the size of the largest input block that can be stored on a single node. In other words, heavy fonts are our goal. If the sharding spans two data blocks, it is almost impossible for any HDFS node to store the two data blocks at the same time, so part of the data in the shard needs to be transmitted to the map task node through the network, which is obviously less efficient than running the entire map task with local data.

In addition, it should be noted that the map task should save the results to the local hard disk, not the HDFS system. Because it produces only intermediate results.

Thank you for your reading, the above is the content of "map fragment size should be the same as block size". After the study of this article, I believe you have a deeper understanding of the problem that map fragment size should be the same as block size, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.