Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the configuration parameters of hadoop YARN". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn what are the hadoop YARN configuration parameters!

The configuration parameters related to MapReduce are divided into two parts, namely JobHistory Server and application parameters. JobHistory can be run on an independent node, while application parameters can be stored in mapred-site.xml as default parameters, or can be specified separately when the application is submitted. Note that if the user specifies the parameters, the default parameters will be overwritten.

The following parameters are all set in mapred-site.xml.

1. MapReduce JobHistory-related configuration parameters

Configure it in the mapred-site.xml of the node where the JobHistory is located.

(1) mapreduce.jobhistory.address

Parameter explanation: MapReduce JobHistory Serve IPC host:port address.

Default value: 0.0.0.0purl 10020

(2) mapreduce.jobhistory.webapp.address

Parameter explanation: MapReduce JobHistory Server Web UI address.

Default value: 0.0.0.0purl 19888

(3) mapreduce.jobhistory.intermediate-done-dir

Parameter explanation: the location of the log generated by the MapReduce job.

Default value: ${yarn.app.mapreduce.am.staging-dir} / history/done_intermediate

(4) mapreduce.jobhistory.done-dir

Parameter explanation: the location where logs managed by MR JobHistory Server are stored.

Default value: ${yarn.app.mapreduce.am.staging-dir} / history/done

2. MapReduce job configuration parameters

It can be configured in the client's mapred-site.xml as the default configuration parameter for the MapReduce job. You can also personalize these parameters when the job is submitted.

The default value of the parameter name indicates the amount of memory consumed by mapreduce.job.name job name mapreduce.job.priorityNORMAL job priority yarn.app.mapreduce.am.resource.mb1536MR ApplicationMaster number of virtual CPU occupied by yarn.app.mapreduce.am.resource.cpu-vcores1MR ApplicationMaster maximum number of failed attempts mapreduce.map.memory.mb1024 number of memory required per Map Task mapreduce.map.cpu.vcores1 number of virtual CPU required per Map Task Maximum number of mapreduce.map.maxattempts4Map Task failed attempts mapreduce.reduce.memory.mb1024 amount of memory required per Reduce Task mapreduce.reduce.cpu.vcores1 number of virtual CPU required per Reduce Task maximum number of failed attempts mapreduce.reduce.maxattempts4Reduce Task enable speculative execution mechanism mapreduce.reduce.speculativefalse enables speculative execution mechanism for Reduce Task queue mapreduce.task.io.sort to which mapreduce.job.queuenamedefault jobs are submitted .mb100 task internal sort buffer size mapreduce.map.sort.spill.percent0.8Map phase threshold for overwriting files (percentage of sort buffer size) number of threads started by mapreduce.reduce.shuffle.parallelcopies5Reduce Task to copy data concurrently

Note that MRv2 renames all configuration parameters in MRv1, but is compatible with the old parameters in MRv1, except that a warning log is printed to remind the user that the parameters are out of date. For the comparison table of new and old parameters of MapReduce, please refer to Java class org.apache.hadoop.mapreduce.util.ConfigUtil.

At this point, I believe you have a deeper understanding of "what are the hadoop YARN configuration parameters?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.