Shulou(Shulou.com)06/02 Report--

When many friends start to build and use YARN clusters, they are easily confused by numerous and complicated configuration parameters: similar parameter names, mixed new and old names, and vague documentation. Especially for those configuration parameters about memory, even if you read the document several times, you can't fully understand the meaning. If you configure it, you will get the wrong object and make mistakes.

If you also encounter the above problems, it does not matter, in this article, I will comb a few difficult to understand YARN memory configuration parameters, and combined with the source code to explain their role and principle, so that you thoroughly understand the meaning of these parameters.

I. the basic structure of YARN

Articles introducing the YARN framework can be found everywhere on the Internet, so I won't elaborate on them here. My previous article, "Analysis and resolution of application JAR package conflicts in YARN environment", also summarized some knowledge points of YARN. You can get the push of this article in the TheFortyTwo background reply number 0x0002. A YARN frame diagram is attached below to facilitate the introduction of our follow-up content:

Figure 1: YARN architecture diagram

2. Carding of memory-related parameters

At first glance, there are many parameters about memory configuration in YARN, but in fact, there are only a few (if you feel that you are actually exposed to more and more confusion than this, because most of the configuration parameters have new and old names, which I will explain later), I have sorted them out and listed them in the following table. Let's take a look at it first. I will explain the meaning of each column in detail later in this section, while I will explain the meaning of each parameter in detail in the next section.

Figure 2: sorting out the memory parameters

Let's explain the columns in the table:

Configuration object: refers to the component for which the parameter works

Parameter name: this does not need to be explained, we all understand

Old parameter name: as we all know, MapReduce has gone through MR1 and MR on YARN; on the large version, while the smaller version has iterated countless times. In the process of the evolution of the version, developers found that the naming of many parameters was not standard, so they modified the parameter names; but in order to ensure the compatibility of the program, they still retained the function of the old parameter names. This is equivalent to the parameters that implement the same function, and there are two different names: the new and the old. For example, the parameters mapreduce.map.java.opts and mapred.map.child.java.opts are actually equivalent. If both the new and old parameters are set, which parameter will actually take effect? The rule of Hadoop is that if the new parameter is set, the new parameter will be used. Otherwise, the value set by the old parameter will be used, regardless of the order in which you set the parameter.

Default value: the default value used by Hadoop if no parameter is set. It is important to note that not all default values for parameters are written in the configuration file (such as mapred-default.xml), such as the parameter mapreduce.map.java.opts, which is obtained from the following code before creating the Map Task:

If (isMapTask) {

UserClasspath = jobConf.get ("mapreduce.map.java.opts"

JobConf.get ("mapred.child.java.opts", "- Xmx200m"))

...

}

As you can see, the priority of this parameter is:

Mapreduce.map.java.opts > mapred.child.java.opts >-Xmx200m

Configuration file: indicates which configuration file is more appropriate if you want to configure this parameter statically (instead of calling API to set the parameter dynamically in the program)

Third, the ultimate explanation of each parameter

Let's explain the function and meaning of each parameter respectively.

Mapreduce.map.java.opts and mapreduce.map.memory.mb

After thinking about it over and over again, I think that these two parameters should be put together to make it easy for everyone to understand, otherwise the separate meeting will make everyone even more confused. The functions of these two parameters are as follows:

Mapreduce.map.java.opts: runs the JVM parameter of the Map task, for example-Xmx specifies the maximum memory size

Mapreduce.map.memory.mb: the maximum amount of memory available for the Container process.

How are these two parameters related? First of all, you need to understand what kind of process Container is (if you want to know more about it, you really need to read my other article, "Analysis and resolution of application JAR package conflicts in YARN environment", reply number 0x0002). To put it simply, Container is actually executing a script file (launch_container.sh), in which a child process of Java is executed, which is the real Map Task.

Figure 3: diagram of Container and Map Task

If you understand this, you will understand that mapreduce.map.java.opts is actually the startup parameters passed to the virtual machine when starting the JVM virtual machine, and the default value-Xmx200m represents the maximum amount of heap memory that the Java program can use. Once this size is exceeded, JVM will throw an Out of Memory exception and terminate the process. Mapreduce.map.memory.mb sets the memory limit of Container, which is read and controlled by NodeManager. When the memory size of Container exceeds this parameter value, NodeManager will be responsible for kill Container. When we analyze the parameter yarn.nodemanager.vmem-pmem-ratio later, we will explain how NodeManager monitors Container memory (including virtual memory and physical memory) and how kill drops Container.

Then, some thoughtful readers may ask these questions:

Q: as mentioned above, Container is a simple script program, and only one JVM program is running in it, so why do you need to set these two parameters separately, instead of simply setting the memory size of JVM is the size of Container?

A: YARN, as a general-purpose computing platform, was designed with programs in various languages running on this platform, not just Java and JVM. So Container is designed as an abstract computing unit, so it has its own memory configuration parameters.

Q: JVM runs as a separate child process of Container, which is different from Container. So is the amount of memory used by JVM limited by the memory size of Container? That is, can the value of the mapreduce.map.java.opts parameter be greater than that of the mapreduce.map.memory.mb?

A: you need to understand how NodeManager manages Container memory. NodeManager has a special monitor thread that monitors the physical and virtual memory usage of all Container at all times to see if each Container exceeds its preset memory size. The Container memory size is calculated by calculating the sum of memory used by all child processes of the Container. As mentioned above, JVM is a child of Container, so the amount of memory used by the JVM process is, of course, included in the amount of memory used by Container. Once a Container uses more memory than its default amount of memory, NodeManager will mercilessly kill it.

Mapreduce.reduce.java.opts and mapred.job.reduce.memory.mb

Similar to the parameters described above, except that these two parameters are specific to Reducer.

Mapred.child.java.opts

This parameter is already an old parameter. In older versions of MR, the JVM memory configuration parameters for Map Task and Reduce Task are not separate and are specified uniformly. In other words, this parameter has actually been divided into mapreduce.map.java.opts and mapreduce.reduce.java.opts, which control Map Task and Reduce Task respectively. However, for compatibility, this parameter is still used in the Hadoop source code, where it is used as described in the above section, and here is a list of priorities:

Mapreduce.map.java.opts > mapred.child.java.opts >-Xmx200m

Yarn.nodemanager.resource.memory-mb

Starting with this parameter, let's look at the configuration items for NodeManager.

This parameter actually sets how much memory NodeManager is prepared to request from this machine, which is used for the allocation and calculation of all Container. This parameter is equivalent to a threshold that limits the maximum amount of memory that NodeManager can use to prevent NodeManager from consuming system memory excessively, resulting in server downtime. This value can be appropriately resized according to the configuration and use of the actual server. For example, our server is the memory configuration of 96GB, on which NodeManager and HBase are deployed, and we allocate 52GB memory for NodeManager.

Yarn.nodemanager.vmem-pmem-ratio and yarn.nodemanager.vmem-check-enabled

The estimation of the parameter yarn.nodemanager.vmem-pmem-ratio is the most confusing. Most of the information found on the Internet comes from the interpretation of official documents, which is not clear enough. Let me explain what this parameter controls with the source code.

First, after NodeManager receives the Container passed by AppMaster, it uses the physical memory size of Container (pmem) * yarn.nodemanager.vmem-pmem-ratio to get the virtual memory size limit of Container, that is, vmemLimit:

Long pmemBytes = container.getResource () .getMemory () * 1024 * 1024L

Float pmemRatio = container.daemonConf.getFloat (YarnConfiguration.NM_VMEM_PMEM_RATIO, YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO)

Long vmemBytes = (long) (pmemRatio * pmemBytes)

NodeManager then monitors Container's pmem (physical memory) and vmem (virtual memory) usage in the monitor thread. If the current vmem is greater than the limit of vmemLimit, or if the memory of olderThanAge (related to JVM memory generation) is greater than the limit, then kill drops the process:

If (currentMemUsage > (2 * vmemLimit)) {

IsOverLimit = true

} else if (curMemUsageOfAgedProcesses > vmemLimit) {

IsOverLimit = true

}

The code for kill process is as follows:

If (isMemoryOverLimit) {

/ / kill the container

EventDispatcher.getEventHandler () .handle (new ContainerKillEvent (containerId, msg)

}

The above control is for virtual memory, and YARN has similar monitoring for the use of physical memory. Readers can explore the source code on their own. The yarn.nodemanager.vmem-check-enabled parameter is very simple, which is the switch for the above monitoring.

The above introduction mentioned vmemLimit, and you may have a question: is vmem here a virtual memory concept at the OS level? Let's take a look at how the source code works.

ContainerMontor is the monitor that monitors the memory usage of each Container in the NodeManager mentioned above, and it is a separate thread. The logic for ContainerMonitor to obtain a single Container memory usage, including physical and virtual memory, is as follows:

Monitor updates the usage of each Container every 3 seconds by:

Look at all the files in the / proc/pid/stat directory to get all the information about each process

Find out all the child processes according to the pid of the current Container, and return the Container as the root node and the child process as the process tree of the leaf node; under the Linux system, the process tree is saved in the ProcfsBasedProcessTree class object

Then get the total amount of virtual memory and physical memory of the current process (Container) from the ProcfsBasedProcessTree class object.

You should know immediately that the amount of memory is obtained through the / proc/pid/stat file, and the amount of memory obtained is the amount of memory for the process and all its child processes. Therefore, the vmem here is the concept of virtual memory at the OS level.

Figure 4: combined schematic diagram of memory parameters

IV. Conclusion

This article gives you an in-depth analysis of several easily confused memory parameters in YARN, and you can find out the analysis methods of similar problems from the point of view of analyzing problems in the article, and combine the documentation with the source code to gain a deeper understanding of the secrets hidden under the framework.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.