Shulou(Shulou.com)06/01 Report--

This article mainly shows you "Hadoop 2.0 how to configure CDH4 MR", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "Hadoop 2.0 in how to configure CDH4 MR" this article.

MR1

Cdh does not recommend using MRv2.0 as a production environment, so mr1 packages are still provided.

Mr1-2.0.0-mr1-cdh5.1.2.tar.gz

When the decompressed copy goes to the original hadoop path, the following file conflicts occur:

[root@h011171 ~] # cp-r hadoop-2.0.0-cdh5.1.2/* / usr/local/hadoop/ cp: overwrite `/ usr/local/hadoop/bin/rcc'? Y cp: overwrite `/ usr/local/hadoop/bin/hadoop'? Y cp: overwrite `/ usr/local/hadoop/cloudera/cdh_version.properties'? Y cp: overwrite `/ usr/local/hadoop/cloudera/apply-patches'? Y cp: overwrite `/ usr/local/hadoop/cloudera/build.properties'? Y cp: overwrite `/ usr/local/hadoop/cloudera/CHANGES.cloudera.txt'? Y

All can be covered.

Basically compatible with all mapreduce configurations of cdh4 mr1, including Fair Scheduler / hadoop acl permission settings

Use

. / bin/start-mapred.sh. / bin/stop-mapred.sh

The monitoring and scheduling interface is the same as the original CDH3

MR2 (YARN)

The most basic design idea of MRv2 is to divide the two main functions of JobTracker, namely, resource management and job scheduling / monitoring, into two independent processes. There are two components in the solution: the global ResourceManager (RM) and the ApplicationMaster associated with each application (AM). The "application" here refers to a single MapReduce job or DAG job. RM and NodeManager (NM, one for each node) make up the whole data computing framework. RM is the final decision maker in the system to allocate resources to each application. AM is actually a concrete framework library whose tasks are to negotiate with RM to obtain the resources needed by the application and to cooperate with NM to complete the task of executing and monitoring task.

The architecture is described as follows:

Configuration

Mapred-site.xml configuration

Mapreduce.framework.name yarn mapreduce.jobhistory.address h011200.hebe.grid.sina.com.cn:10020 mapreduce.jobhistory.webapp.address h011200.hebe.grid.sina.com.cn:19888

Yarn-site.xml configuration

Yarn.resourcemanager.resource-tracker.address h011200.hebe.grid.sina.com.cn:8031 yarn.resourcemanager.address # RMtracker address synonymous with jobtrakcer h011200.hebe.grid.sina.com.cn:8032 yarn.resourcemanager.scheduler.address # RMscheduler address synonymous with mv1 jobscheduler h011200.hebe.grid.sina.com.cn:8030 yarn.resourcemanager.admin.address h011200.hebe.grid.sina.com.cn:8033 yarn.resourcemanager.webapp.address # RM The Web address is the same as the default 50030 h011200.hebe.grid.sina.com.cn:8088 Classpath for typical applications of mr1. Yarn.application.classpath $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $YARN_HOME/* $YARN_HOME/lib/* yarn.nodemanager.aux-services mapreduce.shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.local-dirs / data1/hadoop/data/yarn/local yarn.nodemanager.log-dirs / data1/hadoop/data/yarn/log Where to aggregate logs yarn.nodemanager.remote-app-log-dir / var/log/hadoop-yarn/apps yarn.app.mapreduce.am.staging-dir / user

Start

. / sbin/start-yarn.sh

Local RM and remote NM will be started

. / sbin/mr-jobhistory-daemon.sh start historyserver

Start the local historyserver

RM interface

Job history interface

These are all the contents of the article "how to configure CDH4 MR in Hadoop 2.0". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.