Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

Hadoop fully distributed configuration

2025-01-17 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)06/02 Report--

First of all, Hadoop has three operating modes. The first is the stand-alone (or local) running mode: there is no need to run any daemons, all programs are executed on the same JVM. The second is pseudo-distributed mode: the Hadoop daemon runs on the local machine, simulating a small-scale cluster. The third is fully distributed mode, which is commonly referred to as fully distributed mode: the Hadoop daemon runs on a cluster. Here's a brief introduction to a fully distributed configuration: first prepare three machines System public configuration: system information: [root@hadoop1 ~] # cat / etc/redhat-releaseCentOS release 6.7 (Final) [centos@hadoop1 ~] $uname-r2.6.32-573.el6.x86_64 [root@hadoop1 ~] # uname-mx86_64 public operation user: public configuration of centoscentos users in / etc/sudoers: [root@hadoop1 ~] # grep centos / etc/sudoerscentos ALL= (ALL) NOPASSWD: ALL environment variables are as follows: [ Root@hadoop1 ~] # tail-16 / etc/profile#javaJAVA_HOME=/home/centos/download/jdkPATH=$JAVA_HOME/bin:$PATHCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JAVA_HOMEexport CLASSPATH#hadoopHADOOP_HOME=/home/centos/download/hadoopPATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinexport PATH host IP domain name mapping is as follows: [root@hadoop1 ~] # cat / etc/hosts10.0.0.101 hadoop110.0.0.102 hadoop210.0. 0.103 hadoop3 master node IP is: [root@hadoop1 ~] # ifconfig | awk-F'[:] + 'NR==2 {print $4}' 10.0.0.101 slave node IP is: [root@hadoop2 ~] # ifconfig | awk-F'[:] + 'NR==2 {print $4}' 10.0.102 [root@hadoop3 ~] # ifconfig | awk-F'[:] + 'NR==2 {print $4}' 10.0.0. 103 Java version [root@hadoop1 ~] # java-versionjava version "1.8.0y65" Java (TM) SE Runtime Environment (build1.8.0_65-b17) Java HotSpot (TM) 64-Bit Server VM (build25.65-b01 Mixed mode) Hadoop version [root@hadoop1 ~] # hadoop versionHadoop 2.7.3Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git-r baa91f7c6bc9cb92be5982de4719c1c8af91ccffCompiled by root on 2016-08-18T01:41ZCompiled with protoc 2.5.0From source with checksum2e4ce5f957ea4db193bce3734ff29ff4This command was run using/home/centos/download/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar II. Configure ssh first execute the following generated key pair script in the home directory on the three machines In this way, the centos of the three machines can be password-free login, password-free transfer of files and other operations. [centos@hadoop1 ~] $ll createssh.sh-rwxr-xr-x 1 centos centos 51 May 14 11:05createssh.sh [centos@hadoop1 ~] $cat createssh.shesh.sheshlh.sheshlash Binamp Shsshure keygen-t rsa-P''- f ~ / .ssh/id_rsa then execute the following distribution public key [centos@hadoop1 ~] $ll fen.sh-rwxr-xr-x 1 centos centos 142 May 14 12:14fen.sh [centos@hadoop1 ~] in the home directory on the three machines. ] $cat fen.shen.shangqingqinxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Others will not re-enter the password during execution. The purpose of performing SSH is that, for example, when the master node is fully distributed, the master node automatically hangs the log of the slave node and writes data to it, and the key pair of SSH works. # for frequently encountered pits, configure the permissions of .ssh in the home directory of three machines to 700. And to configure the authorized_keys permission in the .ssh directory in the home directory is 644. [centos@hadoop1 ~] $chmod 700.ssh [centos@hadoop1 ~] $chmod 644.ssh / authorized_keys III, the key configuration file content # three machines jointly configure [centos@hadoop1 ~] $grep JAVA_HOME download/hadoop/etc/hadoop/hadoop-env.sh# The only required environment variable isJAVA_HOME. All others are# set JAVA_HOME in this file The so that it iscorrectly defined onexport JAVA_HOME=/home/centos/download/jdk # JAVA environment variable needs to be configured here for [centos@hadoop1 ~] $cat download/hadoop/etc/hadoop/core-site.xml fs.defaultFS hdfs://hadoop1 [centos@hadoop1 ~] $cat download/hadoop/etc/hadoop/hdfs-site.xml dfs.replication replica number. Corresponding to the number of slave nodes [centos@hadoop1 ~] $cat download/hadoop/etc/hadoop/mapred-site.xml mapreduce.framework.name yarn [centos@hadoop1 ~] $cat download/hadoop/etc/hadoop/yarn-site.xml yarn.resourcemanager.hostname hadoop1 yarn.nodemanager. Aux-services mapreduce_shuffle # Note that only the domain name of the slave node [centos@hadoop1 ~] $cat download/hadoop/etc/hadoop/slaveshadoop2hadoop3 is configured here. Start hadoop # the following commands are executed only in the primary node hadoop1! # first execute the formatting command [centos@hadoop1 ~] $hadoop namenode-format # and then start Hadoop [centos @ hadoop1 ~] $start-all.sh # to view the startup process # execution of the master node. The name node will only exist in the master node, including the file name and so on. [centos@hadoop1 ~] $jps2736 SecondaryNameNode2961 Jps2889 ResourceManager2539 NameNode # executes from the slave node, and the data node only exists in the slave node, which is the real place to store the data. [centos@hadoop2 ~] $jps2322 NodeManager2234 DataNode2380 Jps [centos@hadoop3 ~] $jps2481 DataNode2646 Jps2569 NodeManager # temporary files are generated in the / tmp directory under the root directory [centos@hadoop1 ~] $ll / tmp/total 32drwxrwxr-x 3 centos centos 4096 May 1414purl 05 hadoop-centos-rw-rw-r-- 1 centos centos 5 May 1414: 06 Hadoopcentos1 centos centos 5 May 1414: 06hadoop-centos-secondarynamenode.piddrwxr-xr -x 2 centos centos 4096 May 1414 hsperfdata_centosdrwxrwxr-x 06 hsperfdata_centosdrwxrwxr-x 3 centos centos 4096 May 1414 V 06 Jetty_0_0_0_0_50070_hdfs____w2cu08drwxrwxr-x 3 centos centos 4096 May 1414 V 06 Jetty_0_0_0_0_50090_secondary____y6aanvdrwxrwxr-x 4 centos centos 4096 May 1414 v 06 Jetty_hadoop1_8088_cluster____.jtly5k-rw-rw-r-- 1 centos centos 5 May 1414: 06yarn-centos-resourcemanager.pid

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report