Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to avoid OOM on the troubleshooting shuffle reduce side buffer size". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to avoid OOM on the troubleshooting shuffle reduce side buffer size".

The default buffer size on reduce is 48MB shuffle in Spark and shuffle in MR is absolutely different!

Scene:

The task on the map side continuously outputs data, and the amount of data may be very large. However, in fact, the task on the reduce side does not wait until the task on the map side writes all its own data to the disk file before pulling it. If the map side writes a little data, the reduce side task will pull a small part of the data and immediately carry out the following aggregation and the application of operator functions.

It is up to buffer to decide how much data reduece can pull each time. Because the data pulled is first put in the buffer. Then use the heap memory ratio (0.2) allocated by the later executor, hashmap, to carry out the subsequent aggregation and function execution.

What might go wrong with reduce-side buffering (buffer)?

It may appear. The default is 48MB. Perhaps most of the time, the task on the reduce side will not always be full of 48MB of data while pulling and calculating. Probably most of the time, pull a 10m data and calculate it.

Most of the time, there may not be any problems. But sometimes, the amount of data on the map side is very large, and then the writing speed is very fast. All task on the reduce side, when pulled, all reach the maximum limit of their own buffer, buffer, 48m, all filled.

At this point, coupled with the code for the aggregate function executed on your reduce side, a large number of objects may be created. Perhaps, all of a sudden, the memory will not hold up, will OOM. The problem of memory overflow will occur in the memory on the reduce side.

In view of the above possible problems, how should we solve them?

At this point, you should reduce the size of the task buffer on the reduce side. I prefer Dora to fetch it a few times, but each time I can pull the number of each task on the reduce side at the same time. If there is less, it is not easy to cause OOM memory overflow. (for example, it can be adjusted to 12m)

In the actual production environment, we have all encountered this kind of problem. This is a typical performance-for-execution principle. Reduce buffer is small, it is not easy to OOM, but the performance must be degraded, you have to pull more times. It will take more network transmission overhead.

At this time, we can only take the way of sacrificing performance, spark homework, first of all, the first point is to make sure it can run. Share an experience, once wrote a very complex spark homework, after writing the code, within half a month, just can not run, there are a variety of problems, need to troubleshooting. Adjust more than a dozen parameters, including the size of the buffer on the reduce side. Finally, the homework can run. Then consider performance tuning.

Let's talk about the other side of the buffer size on the reduce side, the side of performance tuning:

Let's say that the amount of data output from your Map is not very large, and then the resources of your entire application are also very sufficient. 200x executor, 5x cpu core, 10G memory.

In fact, you can try to increase the buffer size on the reduce side, for example, from 48m to 96m. In that case, the amount of data that reduce task can pull each time is very large. The number of times you need to pull becomes less. For example, it used to be pulled 100 times, but now it can be executed as long as 50 times.

It is helpful to reduce the overhead of network transmission performance, as well as the number of aggregation operations performed on the reduce side.

The final effect should be a certain degree of improvement in performance.

It must be noted that when there are enough resources, do it again.

Spark.reducer.maxSizeInFlight,48spark.reducer.maxSizeInFlight,24 thank you for reading, the above is the content of "how to avoid OOM on the troubleshooting shuffle reduce side buffer size". After the study of this article, I believe you have a deeper understanding of how to avoid OOM on the troubleshooting shuffle reduce side buffer size, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.