Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to configure memory optimization of YARN and MapReduce". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "how to configure YARN and MapReduce memory optimization"!

In Hadoop 2.x, YARN is responsible for managing resources (memory, CPU, etc.) in MapReduce and packaging them into Containers.

Concentrate on the data processing tasks they are good at, eliminating the need for resource scheduling. as shown in the following figure

YARN manages the available compute resources of all machines in the cluster. Based on these resources, YARN schedules resource requests from applications (such as MapReduce), and YARN then allocates Co

Container is the basic unit of processing power in YARN, which encapsulates memory, CPU, etc.

At present, my server situation here: 6 slaves, each: 32G memory, 2*6 core CPU.

Since Hadoop 1.x has JobTracker and TaskTracker, resource management has their implementations, and when mapreduce jobs are executed, resources are divided into map task and reduce task.

The following two parameters set the number of tasks each TaskTracker can run:

mapred.tasktracker.map.tasks.maximum

6

mapred.tasktracker.reduce.tasks.maximum

4

The maximum number of reduce tasks a task tracker can run simultaneously

However, in hadoop 2.x, Yarn architecture is introduced for resource management, NodeManager is run on each node to allocate node resources, and slots are no longer distinguished from Map slots and Reduce slots as in 1.x. Above Yarn Container is the smallest unit of allocation of resources.

Memory allocation configuration for Yarn cluster is configured in yarn-site.xml file:

yarn.nodemanager.resource.memory-mb

22528

Available memory per node in MB

yarn.scheduler.minimum-allocation-mb

1500

A single task can request a minimum of memory, default 1024MB

yarn.scheduler.maximum-allocation-mb

16384

Maximum memory for a single task, default 8192MB

Since my Yarn cluster also needs to run Spark tasks, and Spark Worker memory needs to be relatively large, so we need to increase the maximum memory of a single task (default is 8G).

And Mapreduce's task memory configuration:

mapreduce.map.memory.mb

1500

Physical memory limit per Map task

mapreduce.reduce.memory.mb

3000

Physical memory limit per Reduce task

mapreduce.map.java.opts

-Xmx1200m

mapreduce.reduce.java.opts

-Xmx2600m

mapreduce.map.memory.mb: Memory of each map task, which should be greater than or equal to the minimum memory of the Container.

Each slave can run mapreduce.reduce.java.opts on map data as configured above.

mapreduce.map.java.opts / mapreduce.map.memory.mb

=0.70~0.80

mapreduce.reduce.java.opts / mapreduce.reduce.memory.mb

=0.70~0.80

In the yarn container mode, JVM processes run in containers, mapreduce. {map|reduce}.java.opts 能够通过Xmx设置JVM最大的heap的使用，

Memory.mb is typically set to 0.75 times

Then reserve some space for java,scala code, etc.

At this point, I believe everyone has a deeper understanding of "how to configure YARN and MapReduce memory optimization," so let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.