Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to configure the Hadoop environment, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand.

The Hadoop environment needs to be configured after installation.

Hadoop components are configured through XML, core-site.xml is used to configure the properties of Common components, hdfs-site.xml files are used to configure HDFS properties, mapred-site.xml files are used to configure MapReduce properties, and hdoop supports independent mode, pseudo-distribution mode and full distribution mode. In different modes, the configuration of each attribute:

Component property independent mode pseudo distribution mode commonfs.default.name file:///( default) hdfs://localhost/hdfs://namenodehdfsdfs.replicationN/A13mapreducemapred.job.tracker (hadoop2 does not need to be configured) local (default) localhost:8021

Jobtracker:8021

Hadoop log configuration: adding export HADOOP_LOG_DIR=/usr/lib/hadoop/logs to hadoop-env.sh

My hadoop2 configuration is as follows:

Core-site.xml

Fs.default.name hdfs://localhost/ hadoop.tmp.dir / usr/lib/hadoop/tmp/

Hdfs-site.xml:

Dfs.default.name / usr/lib/hadoop/logs/ dfs.replication 1 dfs.permissions false

Mapred-site.xml:

Mapreduce.framework.name Yarn

Yarn-site.xml:

Yarn.resourcemanager.resource-tracker.address localhost:8990 yarn.resourcemanager.scheduler.address localhost:8991 yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanage.scheduler.capacity.CapacityScheduler yarn.resourcemanager.address Localhost:8993 yarn.nodemanager.local-dirs / usr/lib/hadoop/tmp/node yarn.nodemanager.address localhost:8994 yarn.nodemanager.resource.memory-mb 102040 Yarn.nodemanager.remote-app-log-dir / usr/lib/hadoop/tmp/app-logs yarn.nodemanager.log-dirs / usr/lib/hadoop/tmp/node yarn.nodemanager.aux-services mapreduce_shuffle

Slaves:

Localhost

When you are done, execute hadoop namenode-fromat to format the hdfs file system.

Thank you for reading this article carefully. I hope the article "how to configure the Hadoop Environment" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.