Shulou(Shulou.com)06/03 Report--

HA (high availability) HA uses distributed log management method 1. There is a problem with the Namenode and the entire cluster will not be available. Configure two namenode:Active namenode,standby namenode 2. Implementation method 1. The metadata stored in the memory of the two namenode is synchronized, and the mirror file is read when the namenode starts. two。 Secure distributed storage log files for editing logs, stored on odd nodes of 2n+1. (n nodes were successfully written, and log writes were successful. ) Zookeeper monitor monitors two namenode, one of which has a problem and fails over. Zookeeper has higher requirements for time synchronization (ntp time synchronization) 3. How does the client know which namenode to access using the proxy agent isolation mechanism use the password-less login installation configuration 1. 1 between the two namenode of sshfence. Basic environment configuration node1node2node3node1node2192.168.103.26192.168.103.27192.168.103.28192.168.103.29192.168.103.30namenodenamenodedatanodedatanodedatanodeDFSZKFailoverControllerDFSZKFailoverControllerjournalnodejournalnodejournalnodeQuorumPeerMainQuorumPeerMainQuorumPeerMain configuration mapping between hostname and IP vim / etc/hosts 192.168.103.26 node1 192.168.103.27 node2 192.168.103.28 node3 192.168.103.29 node4 192.168.103.30 node5 configuration secret-free login between nodes Node1 ssh-kengen-t rsa-P''generate id_rsa under ~ / .ssh / directory Id_rsa.put key ssh-copy-id-I ~ / .ssh/id._rsa.pub (node1,node2,node3,node4.node5) Node2 operation synchronizes with node1 configuration time Node1 serves as ntp server 1. Yum install ntp-y (all nodes) 2. Node1 vim / etc/ntp.conf server 210.72.145.44 # China National time Center server 127.127.1.0 # provides services for local area network users fudge 127.127.1.0 stratum 10 systemctl start ntpd 3.node3jue Node4Node5Node2. Install hadoop 1.tar-zxvf jdk-8u171-linux-x64.tar.gz-C / mv jdk1.8.0_171/ jdk tar-zxvf hadoop-2.7.7.tar.gz-C / mv hadoop-2.7.7/ Hadoop tar-zxvf zookeeper-3.4.10.tar.gz-C / mv zookeeper-3.4.10 zookeeper 3.vim / etc/profile export JAVA_HOME=/jdk export HADOOP_HOME=/Hadoop export ZOOKEEPER_HOME=/zookeeper export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin source / etc/profile scp / etc/profile node2:/etc/ node3 Node4,node5 scp-r / jdk node2:/etc node3,node4,node5 4. Configure zookeeper to enter the zookeeper directory, create a zkdata directory and create a myid file. The value in the node3,node4,node5 file is 1Magne2 scp-r / zookeeper node2:/etc node3,node4,node5 to modify the myid file of the zookeeper node 5. Install hadoop (key!!) 1. Hadoop-env.sh export JAVA_HOME = / jdk 2. Core-site.xml fs.defaultFS hdfs://ns1 ha.zookeeper.quorum node3:2181,node4:2181 Node5:2181 hadoop.tmp.dir / hadoop/tmp 3. Hdfs.site.xml dfs.nameservices ns1 dfs.ha.namenodes.ns1 nn1 Nn2 dfs.namenode.rpc-address.ns1.nn1 node1:8020 dfs.namenode.rpc-address.ns1.nn2 node2:8020 dfs.ha.namenode.http-address.ns1.nn1 node1:50070 dfs.ha.namenode.http-address.ns1.nn2 node2: 50070 dfs.namenode.shared.edits.dir qjournal://node3:8485 Node4:8485 Node5:8485/ns1 dfs.journalnode.edits.dir / journalnode dfs.client.failover.proxy.provider.ns1 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence dfs.ha.fencing.ssh.private-key-files / home/root/ .ssh / id_rsa dfs.ha.automatic-failover.enabled true vim slaves node3 node4 node5 4. Start node3,node4,node5 hadoop-daemon.sh start journalnode zkServer.sh start node1 hdfs namenode-format scp-r / Hadoop/tmp node2:/Hadoop/ hdfs zkfc-formatZK start-dfs.sh 5. Verify that HDFS HA checks the status of node1 and node2 namenode through the browser! [] (https://cache.yisu.com/upload/information/20200310/72/146158.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)! [] (https://cache.yisu.com/upload/information/20200310/72/146160.jpg?x-oss-process=image/watermark,size_16, Text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=) hadoop-daemon.sh stop namenode! [] (https://cache.yisu.com/upload/information/20200310/72/146161.jpg?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.