Shulou(Shulou.com)06/02 Report--

I. description of the problem

Previously, because java.lang.OutOfMemoryError: unable to create new native thread set the Xss parameter, see http://zouqingyun.blog.51cto.com/782246/1879975

The exception still occurs in nodeManager, as well as in the task of map-reduce

2. Some phenomena

Ran a map-reduce task, which dealt with small files, and finally generated more than 20, 000 map tasks. Many tasks in this job appear java.lang.OutOfMemoryError: unable to create new native thread, observe some tasks of this job, and find that the thread stack of this task continues to grow, and finally there are more than 7000 thread, which finally leads to java.lang.OutOfMemoryError: unable to create new native thread, because each map task allocates memory of 800m map ThreadStackSize, which is 1024k by default, resulting in memory exhaustion. Continue the output in the thread stack of the task:

"Thread-3689" daemon prio=10 tid=0x00007fb6bf364000 nid=0x2331 in Object.wait () [0x00007fb5b9b94000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait (Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run (DFSOutputStream.java:638)-locked (a java.util.LinkedList) "Thread-3688" daemon prio=10 tid=0x00007fb6bf362000 nid=0x10a9 in Object.wait () [0x00007fb5b9c95000] java.lang.Thread.State: TIMED_ WAITING (on object monitor) at java.lang.Object.wait (Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run (DFSOutputStream.java:638)-locked (a java.util.LinkedList) "Thread-3687" daemon prio=10 tid=0x00007fb6bf35a800 nid=0xf23 in Object.wait () [0x00007fb5b9d96000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait (Native Method) at org.apache .hadoop.hdfs.DFSOutputStream $DataStreamer.run (DFSOutputStream.java:638)-locked (a java.util.LinkedList) "Thread-3686" daemon prio=10 tid=0x00007fb6bf358800 nid=0xde9 in Object.wait () [0x00007fb5b9e97000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait (Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run (DFSOutputStream.java:638)

Third, guess

1. The exception of nodemanager may be related to this. When all tasks in the map-reduce are scheduled to one machine (about 40 container), the tasks in each container generate 7000 thread (generate many small files?) . Resulting in depletion of max user processes (262144). But when nodemanger needs new thread, java.lang.OutOfMemoryError: unable to create new native thread appears. (ps did run regularly on this task yesterday.)

2. It may be a memory overflow problem in some parts of hadoop/yarn. See a similar question. Https://issues.apache.org/jira/browse/YARN-4581

IV. Postscript

Hadoop handles a large number of small files, use org.apache.hadoop.mapreduce.lib.input.CombineTextInputFormat, and set mapreduce.input.fileinputformat.split.maxsize = 5147483648

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.