Shulou(Shulou.com)05/31 Report--

The purpose of this article is to share with you what are the knowledge points of entry configuration in hadoop. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Environment

I use XUbuntu14.04 64-bit, built under the virtual machine.

Install JDK and configure environment variables

Configure classpath to modify environment variables for all users

Sudo vim / etc/profile# add # set java environmentexport JAVA_HOME= {path} export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/bin source/etc/profile # # at the end of the file to make the environment variable take effect

Configure SSH without password to log in

Create a hidden folder .ssh in the user root directory (none need to be created)

Mkdir. Sshssh-keygen-t dsa-P''- f ~ / .ssh/id_dsacat ~ / .ssh/id_dsa.pub > > ~ / .ssh/authorized_keys

Verify the ssh

Shh localhost

Install hadoop (I use 2. 7)

Extract the folder

Sudo tar-zxvf hadoop-2.7.1.tar.gz-C / usr/local/hadoop # specifies the directory, and you can choose where to put it according to your preference.

Create a file directory for hadooop work

Set the environment variable (enter $HADOOP_HOME/etc/hadoop)

Sudo vim / etc/profileexport HADOOP_HOME= {path} # path is followed by export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Configuration file

Configure hadoop-env.sh

Export JAVA_HOME=$ {JAVA_HOME} # JAVA_HOME JDK path

Configure core-site.xml (a note on the Hadoop configuration item, although you only need to configure fs.defaultFS and dfs.replication to run (as is the case with official tutorials), if you do not configure the hadoop.tmp.dir parameter, the default temporary directory is / tmp/hadoo-hadoop, and this directory may be cleaned up by the system upon restart, resulting in a re-execution of format. So we set it up and also specify dfs.namenode.name.dir and dfs.datanode.data.dir, otherwise we may make an error in the next steps. Shay hostname. If you don't know your hostname, you can cat it / etc/hostname)

Hadoop.tmp.dir file:/usr/local/hadoop/tmp fs.defaultFS hdfs://shay:9000

Configure hdfs-site.xml

Dfs.replication 1 dfs.namenode.name.dir file:/usr/local/hadoop/tmp/dfs/name dfs.datanode.data.dir file:/usr/local/hadoop/tmp/dfs/data

Configure mapred-site.xml

Mapreduce.framework.name yarn

Configure yarn-site.xml

Yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.hostname shay

Configure yarn-env.sh

Export JAVA_HOME=$ {JAVA_HOME}

Hadoop start

Perform namenode formatting

Hdfs namenode-format

Start the daemon

Start-dfs.sh # NameNode and DataNode daemons start-yarn.sh # ResourceManager and NodeManager daemons

Verification

Jps, thank you for your reading! On "what are the entry configuration knowledge points in hadoop" this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, you can share it out for more people to see it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.