Shulou(Shulou.com)06/03 Report--

Batch task processing performed through Hive failed. The error log reported in Spark is as follows:

[plain] view plain copy

ERROR: Failed to monitor Job [3] with exception 'java.lang.IllegalStateException (RPC channel is closed.)'

Java.lang.IllegalStateException: RPC channel is closed.

At com.google.common.base.Preconditions.checkState (Preconditions.java:145)

At org.apache.hive.spark.client.rpc.Rpc.call (Rpc.java:277)

At org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.run (SparkClientImpl.java:584)

At org.apache.hive.spark.client.SparkClientImpl.run (SparkClientImpl.java:151)

At org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobStatus.getSparkJobInfo (RemoteSparkJobStatus.java:147)

Through the ResourceManager of Yarn, you can see the error log as follows:

[plain] view plain copy

Diagnostics: Container [pid=31880,containerID=container_1528360247633_0013_01_000001] is running beyond physical memory limits.

Current usage: 1.0 GB of 1 GB physical memory used; 2.8 GB of 2.1 GB virtual memory used. Killing container.

The message "1.0 GB of 1 GB physical memory used; 2.8 GB of 2.1 GB virtual memory used" indicates that the problem is caused by insufficient memory, so the memory of the Yarn container needs to be optimized.

1. Adjust the maximum available memory for each container node

This is determined according to the actual situation of the server currently in use. In addition, you need to consider reserving available memory for the operating system. For example, the memory of the current server is 16G, which can be configured to 12G, and the remaining 4G is reserved for the operating system:

[html] view plain copy

Yarn.nodemanager.resource.memory-mb

12000

Note: configured in yarn-site.xml

two。 Adjust the minimum memory that can be applied for by a single container

The adjustment parameter is yarn.scheduler.minimum-allocation-mb. This value and the maximum storage of a single container mentioned in point 1 can be used to measure the maximum number of containers that a single server can run. The value is now adjusted to 2G:

[html] view plain copy

Yarn.scheduler.minimum-allocation-mb

2048

Note: configured in yarn-site.xml

3. Increase the memory available for Map and Reduce

In the current example, 2G of memory is set for each container (through the parameter yarn.scheduler.minimum-allocation-mb=2048), so we allocate 4G of memory to the Map task container and 8G of memory to the Reduce task container:

[html] view plain copy

Mapreduce.map.memory.mb

4096

Mapreduce.reduce.memory.mb

8192

Note: configured in mapred-site.xml

4. Optimize JVM memory for Map and Reduce tasks

Each Container runs the Map and Reduce tasks, so the JVM heap size should be set below the Map and Reduce memory defined above so that they fall within the range of Container memory allocated by YARN.

[html] view plain copy

Mapreduce.map.java.opts

-Xmx3072m

Mapreduce.reduce.java.opts

-Xmx6144m

Note: configured in mapred-site.xml

5. Optimize the proportion of virtual memory

The upper limit of virtual memory (physical + paging memory) for each Map and Reduce task is determined by the virtual memory ratio of each allowed YARN container. This is set by the following configuration, with a default value of 2.1:

[html] view plain copy

Yarn.nodemanager.vmem-pmem-ratio

2.1

Note: configured in yarn-site.xml

If you want to know big data's learning route, if you want to learn big data knowledge and need free learning materials, you can add group: 784789432. Welcome to join us. Every day, a live broadcast will be held at 3 pm to share basic knowledge, and at 20:00 in the evening, a live broadcast will be held to share the actual combat of big data project.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.