Shulou(Shulou.com)05/31 Report--

This article mainly explains "what factors determine the number of Task of Map and Reduce". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "which factors determine the number of Task of Map and Reduce".

Unspecified, the number of Map and Reduce Task is determined by several factors:

1. The number of blocks of input data, the number of blocks of Map output data (the number of Reduce can be set), and a block is generally processed by a task (generally, that is, whether the file format supports sharding)

two。 Enter the number of files for the data. A file requires at least 1 task to process (at least, a file may be stored in multiple file blocks)

3. Whether the data is compressed, whether the compression format can be split (Splitable), if not, even if the file is larger than 64m, it can only be processed by one task (does not support sharding, sequential reading)

Thank you for your reading, the above is the content of "what factors determine the number of Task of Map and Reduce". After the study of this article, I believe you have a deeper understanding of what factors determine the number of Task of Map and Reduce, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.