What is the method of mapreduce performance optimization

This article mainly explains "what is the method of mapreduce performance optimization". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the method of mapreduce performance optimization".

The number of map task seriously affects the performance of mr. There are a lot of materials on the Internet, which explain the principles and methods, but I will not elaborate on them here. I have summed up two revision methods, which I hope will be helpful to you.

Method 1: modify the blocksize of hdfs, which is undesirable and requires a restart of the cluster. The key is that the previous data is still the same size as the previous blocksize, and the new data will be the size of the new blocksize. For example, it used to be 128m, which was modified to 512m. The previous data has not changed, and the blocksize is still 128m.

Method 2: recommend the method. Code settings, the most flexible, according to different requirements, set different values. Add FileInputFormat.setMinInputSplitSize to the code that sets the size of mapred.min.split.size in bytes, such as my code:

Job.setMapOutputValueClass (IntWritable.class)

Job.setNumReduceTasks (1)

/ / set the minimum shard to 512m

FileInputFormat.setMinInputSplitSize (job, 1024,1024,512)

FileInputFormat.addInputPath (job, new Path ("/ usr/keyword/input"))

Supplementary note:

TextInputFormat.setMinInputSplitSize (job,1024L); / / set the minimum shard size

TextInputFormat.setMaxInputSplitSize (job,1024 × 1024 × 10L); / / set the maximum shard size

Thank you for your reading, the above is the content of "what is the method of mapreduce performance optimization". After the study of this article, I believe you have a deeper understanding of what the method of mapreduce performance optimization is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!