Shulou(Shulou.com)06/03 Report--

Hadoop 2.7.7 installation (test environment deployment) hadoop2.x deployment

System environment (censtos 6.5):

172.16.57.97 namenodeyw172.16.57.98 datanodeyw1172.16.57.238 datanodeyw2

Software download:

Hadoop

Wget http://mirrors.shu.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

Zookeeper3.4.10

Wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

Hbase2.0.2

Wget http://mirror.bit.edu.cn/apache/hbase/2.0.2/hbase-2.0.2-bin.tar.gz

Jdk1.8.0_92:

All cluster server ssh ports must be the same

I. Environmental configuration

1. Three servers modify hostname and system parameters (limit.conf)

# hostname namenodeyw#vim / etc/sysconfig/networkNETWORKING=yesHOSTNAME=namenodeyw#vim / etc/security/limits.conf* soft nofile 102400 * hard nofile 102400 * soft nproc 65536 * hard nproc 65536

Check whether / etc/security/limits.d/90-nproc.conf exists. If so, you need to add the configuration.

* soft nproc 6553

2. Create a hadoop user and set the hadoop user password (set the home directory to the one with a large partition)

# useradd-d / opt/hadoop-g hadoop-m hadoop (# assign hadoop home directory to / opt/, specify hadoop user group and login user directory) # passwd hadoop (# set password)

3. Do key-free login under three hadoop users.

Execute ssh-keygen-t rsa to enter the .ssh directory: cd / opt/hadoop/.sshssh-copy-id-I id_rsa.pub "- p 45685 hadoop@172.16.57.97" (pairwise ssh key-free) ssh- p ssh port hostname (mutual login authentication)

4. Install jdk on three servers, version jdk1.8.0_92

Extract the installation package in / usr/java/

Configure the environment variable / etc/profile and .bash _ profile under hadoop users

# jdkexport JAVA_HOME=/usr/java/jdk1.8.0_92export CLASSPATH=.:$JAVA_HOME/lib/tools.jarexport PATH=$JAVA_HOME/bin:$PATH

Execution of source / etc/profile takes effect

5. Set / etc/selinux/config to disable to restart the system

# vim / etc/selinux/configSELINUX=disabled

6. Configure firewall access.

7. Add hostname and IP address mapping on each of the three servers (synchronized by root users)

# vim / etc/hosts172.16.57.97 namenodeyw172.16.57.98 datanodeyw1172.16.57.238 datanodeyw2

Second, install hadoop 2.7.7

1. After logging in with hadoop users, extract the hadoop package under the current directory, and then configure the environment variables.

Cd / opt/hadooptar zxf hadoop-2.7.7.tar.gz

Configure the environment variable / etc/profile and the .bash _ profile file under hadoop

# hadoop 2.7.7export HADOOP_HOME=/opt/hadoop/hadoop-2.7.7export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

Execute the command source / etc/profile to make the command effective

2. Hadoop configuration modification

Refer to attachment settings

3. Configuration modification

Vi hadoop-env.shexport HADOOP_SSH_OPTS= "- p 45685" # if the ssh port is not 22 by default, you need to add this line export JAVA_HOME=/usr/java/jdk1.8.0_92

4. Copy the modified hadoop-2.7.7 directory on namenode to other nodes

$scp-r-P 45685 hadoop-2.7.7 datanodeyw1:/opt/hadoop/$scp-r-P 45685 hadoop-2.7.7 datanodeyw2:/opt/hadoop/

5. Initialize the settings on namenode

$cd / opt/hadoop/hadoop-2.7.7/bin$./hadoop namenode-format

6. Execute the startup command:

Jps View process is empty before starting hadoop

The schedule after executing start-all.sh is as follows:

The services started on namenode are as follows

[hadoop@namenodeyw sbin] $jps1507 Jps1079 ResourceManager906 SecondaryNameNode731 DataNode1196 NodeManager589 NameNode [hadoop@namenodeyw sbin] $jps-l1524 sun.tools.jps.Jps1079 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager906 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode731 org.apache.hadoop.hdfs.server.datanode.DataNode1196 org.apache.hadoop.yarn.server.nodemanager.NodeManager589 org.apache.hadoop.hdfs.server.namenode.NameNode

The services started on datanodeyw1 are:

[hadoop@datanodeyw1 ~] $jps24103 NodeManager24346 Jps23965 DataNode [hadoop@datanodeyw1 ~] $jps-l24103 org.apache.hadoop.yarn.server.nodemanager.NodeManager24361 sun.tools.jps.Jps23965 org.apache.hadoop.hdfs.server.datanode.DataNode

The services started on datanodeyw2 are:

[hadoop@datanodeyw2 hadoop] $jps23664 DataNode23781 NodeManager23902 Jps [hadoop@datanodeyw2 hadoop] $jps-l23664 org.apache.hadoop.hdfs.server.datanode.DataNode23781 org.apache.hadoop.yarn.server.nodemanager.NodeManager23914 sun.tools.jps.Jps

7. Verification of creating directories and uploading files on hadoop

Hadoop fs-mkdir-p / data/testhadoop fs-put hadoop-2.7.7.tar.gz / data/test/hadoop fs-ls / data/test # check whether the file is uploaded successfully

8. Web page query

Http://172.16.57.97:8088/cluster/cluster # hadoop cluster http://172.16.57.97:50070/dfshealth.html#tab-overview # go to browse the file to see if the uploaded file exists

Third, install Zookeeper3.4.10

1. Extract the downloaded Zookeeper3.4.10 to the / opt/hadoop directory on namenode.

2. Modify the environment variable / etc/profile and the. bash_profile file under the hadoop user on the three servers and make it effective.

3. Create directories on three servers

Mkdir-p / opt/hadoop/zookeeper/ {data,dataLog}

4. Create myid files on three servers

Namenodeyw: echo "1" > > / opt/hadoop/zookeeper/data/myiddatanodeyw1: echo "2" > > / opt/hadoop/zookeeper/data/myiddatanodeyw2: echo "3" > > / opt/hadoop/zookeeper/data/myid

Description: the new directory above may not be the same as mine, and the number in myid can also be different, as long as it corresponds to the configuration of the following zoo.cfg, but it doesn't hurt to build the same.

5. Modify zookeeper configuration zoo.cfg ()

Enter / opt/hadoop/zookeeper-3.4.10/conf

$cd / opt/hadoop/zookeeper-3.4.10/conf$cp-r zoo_sample.cfg zoo.cfg$vim zoo.cfgdataDir=/opt/hadoop/zookeeper/datadataLogDir=/opt/hadoop/zookeeper/dataLogserver.1=namenodeyw:2888:3888server.2=datanodeyw1:2888:3888server.3=datanodeyw2:2888:3888

6. Copy the modified zookeeper-3.4.10 directory on namenode to other nodes

$scp-r-P 45685 zookeeper-3.4.10 datanodeyw1:/opt/hadoop/$scp-r-P 45685 zookeeper-3.4.10 datanodeyw2:/opt/hadoop/

7. Start the zookeeper cluster respectively

Zookeeper usage:

Usage: / opt/hadoop/zookeeper-3.4.10/bin/zkServer.sh {start | start-foreground | stop | restart | status | upgrade | print-cmd}

two。 Command:

/ opt/hadoop/zookeeper-3.4.10/bin/zkServer.sh start # start / opt/hadoop/zookeeper-3.4.10/bin/zkServer.sh stop # stop / opt/hadoop/zookeeper-3.4.10/bin/zkServer.sh status # check status (verify that the result is normal)

Description: dataDir and dataLogDir need to be created by themselves, and the directory can be made by themselves. The 1 in server.1 needs to correspond to the value in the myid file in the dataDir directory on the namenodeyw machine. The 2 in server.2 needs to correspond to the value in the myid file in the dataDir directory on the datanodeyw1 machine. The 3 in server.3 needs to correspond to the value in the myid file in the dataDir directory on the datanodeyw2 machine. Of course, you can use the value as long as you correspond to it. The port numbers of 2888 and 3888 can also be used freely, because there is nothing to use them on different machines.

It is called.

8. Verify the normal result of zookeeper:

IV. Hbase installation and configuration

1. Unpack the downloaded hbase2.0.2.tar.gz installation package to / opt/hadop/ on the namenode server.

2. Each node modifies the environment variable / etc/profile and the. bash_profile file under the hadoop user and makes it effective

3. Create a folder

Mkdir-p / opt/hadoop/hbase/ {tmp,pids}

4. Modify the hbase configuration on nodename (see attachment)

Vim / opt/hadoop/hbase-2.0.2/conf/hbase-env.shexport HBASE_SSH_OPTS= "- p 45685" # if the ssh port is not 22 by default, you need to add this line export JAVA_HOME=/usr/java/jdk1.8.0_92export HADOOP_HOME=/opt/hadoop/hadoop-2.7.7export ZOOKEEPER_HOME=/opt/hadoop/zookeeper-3.4.10export HBASE_HOME=/opt/hadoop/hbase-2.0.2export HBASE_PID_DIR=/root/hbase/pidsexport HBASE_MANAGES_ZK=false

Hbase-site.xml

Hbase.rootdirhdfs://namenodeyw:9000/hbaseThe directory shared byregion servers.hbase.zookeeper.property.clientPort2181Property from ZooKeeper'sconfig zoo.cfg. The port at which the clients will connect.zookeeper.session.timeout120000hbase.zookeeper.quorumnamenodeyw,datanodeyw1,datanodeyw2hbase.tmp.dir/opt/hadoop/hbase/tmphbase.cluster.distributedtrue

Regionservers (according to the configuration of regionservers, if datanodeyw1 is missing in the file, start hbase,datanodeyw1 as a hmaster service on datanodeyw1)

Namenodeywdatanodeyw2

5. If the system ssh port is not 22, you need to modify the configuration.

Vim hbase-env.shexport HBASE_SSH_OPTS= "- p 45685"

6. Synchronize the modified hbase-2.0.2 directory on namenodeyw to other nodes / opt/hadoop path

$scp-r-P 45685 hbase-2.0.2 datanodeyw1:/opt/hadoop/$scp-r-P 45685 hbase-2.0.2 datanodeyw2:/opt/hadoop/

7. Start the hbase service on datanodeyw1 (according to the configuration of regionserversonline, if datanodeyw1 is missing in the file, start the hbase file on datanodeyw1, then the node is used as a hmaster service)

Cd / opt/hadoop/hbase-2.0.2/bin./start-hbase.sh

8. Verify that you can view the process after startup.

Datanodeyw1:

[hadoop@datanodeyw1 bin] $jps4835 HMaster # hbase master node 6667 Jps25675 DataNode30862 QuorumPeerMain # zookeeper25807 NodeManager

Datanodeyw2:

[hadoop@datanodeyw2 home] $jps24805 HRegionServer # hbase daemon 24485 QuorumPeerMain # zookeeper24983 Jps24057 DataNode24171 NodeManager

Namenodeyw:

[hadoop@namenodeyw ~] $jps2832 NodeManager2387 DataNode2248 NameNode4072 HRegionServer4251 Jps3596 QuorumPeerMain # zookeeper2716 ResourceManager2558 SecondaryNameNode

9. Query status of web page

Http://172.16.57.98:16010/master-status

5. Add zabbix monitoring

1. Add crontab-e under hadoop users on any hadoop node

* / 5 * / opt/hadoop/hadoop-2.7.7/bin/hadoop dfsadmin-report > / tmp/hadoop.status*/5 * / bin/echo "status" | / opt/hadoop/hbase-2.0.2/bin/hbase shell > / tmp/hbase.status

two。 Modify zabbix configuration file zabbix_agentd.conf.d/hadoop.conf

Hadoop1.x

Vi zabbix_agentd.conf.d/hadoop.conf# status Monitoring keyUserParameter=dfs.status,cat / tmp/hadoop-status | grep "dead" | cut-d''- f 6UserParameterparts hbase.statusCat / tmp/hbase-status | grep "server" | cut-d'- f 1

Hadoop2.7x

Vi zabbix_agentd.conf.d/hadoop.conf# status Monitoring keyUserParameter=dfs.status,cat / tmp/hadoop.status | grep "Live datanodes" | awk-F "[()]"'{print $2} 'UserParameter=hbase.status,cat / tmp/hbase.status | grep "dead" | cut-d','- f 4 | cut-d''- f 2

3. Restart the zabbix-agentd service

4. Add monitoring items to zabbix

The dfs.status monitoring parameter is the alarm after the datanode node dies.

Zabbix Monitoring add

There is dead node processing in dfs

Http://172.16.57.97:50070/dfshealth.html#tab-overview (# namenode)

Click Dead Nodes on the page to enter:

Startup mode:

Hadoop-daemon.sh start datanode

The hbase.status monitoring parameter is the alarm after the hbase process node dies.

Zabbix Monitoring add

There is dead node processing in hbase

Http://172.16.57.98:16010/master-status # (the link address is the server IP and port where the hmaster process resides)

The page shows that the hostname in Dead Region Servers is the point of failure server.

Startup mode:

Hbase-daemon.sh start regionserver

6 、 FAQ

1 execute start-all.sh error report

[hadoop@namenodeyw sbin] $. / start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

18-09-10 12:12:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

Starting namenodes on [namenodeyw]

Namenodeyw: starting namenode, logging to / opt/hadoop/hadoop-2.7.7/logs/hadoop-hadoop-namenode-namenodeyw.out

Namenodeyw: starting datanode, logging to / opt/hadoop/hadoop-2.7.7/logs/hadoop-hadoop-datanode-namenodeyw.out

Datanodeyw2: starting datanode, logging to / home/hadoop/hadoop-2.7.7/logs/hadoop-hadoop-datanode-datanodeyw2.out

Datanodeyw1: starting datanode, logging to / opt/hadoop/hadoop-2.7.7/logs/hadoop-hadoop-datanode-datanodeyw1.out

Datanodeyw2: [Fatal Error] core-site.xml:24:5: The markup in the document following the root element must be well-formed.

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to / opt/hadoop/hadoop-2.7.7/logs/hadoop-hadoop-secondarynamenode-namenodeyw.out

18-09-10 12:12:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

Starting yarn daemons

Starting resourcemanager, logging to / opt/hadoop/hadoop-2.7.7/logs/yarn-hadoop-resourcemanager-namenodeyw.out

Datanodeyw2: starting nodemanager, logging to / home/hadoop/hadoop-2.7.7/logs/yarn-hadoop-nodemanager-datanodeyw2.out

Namenodeyw: starting nodemanager, logging to / opt/hadoop/hadoop-2.7.7/logs/yarn-hadoop-nodemanager-namenodeyw.out

Datanodeyw1: starting nodemanager, logging to / opt/hadoop/hadoop-2.7.7/logs/yarn-hadoop-nodemanager-datanodeyw1.out

Datanodeyw2: [Fatal Error] core-site.xml:24:5: The markup in the document following the root element must be well-formed.

Incorrect configuration file format results in:

2. The problem of error in zookeeper status query

[hadoop@namenodeyw conf] $/ opt/hadoop/zookeeper-3.4.10/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: / opt/hadoop/zookeeper-3.4.10/bin/../conf/zoo.cfg

Error contacting service. It is probably not running.

Resolution: check whether the myid and zoo.cfg files are misconfigured

Normal result output:

2. Problem description

The question was forwarded to: https://www.cnblogs.com/zlslch/p/6418248.html

Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable (44)

Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

[hadoop@djt002 native] $pwd

/ usr/local/hadoop/hadoop-2.6.0/lib/native

[hadoop@djt002 native] $ls

Libhadoop.a libhadooppipes.a libhadoop.so libhadoop.so.1.0.0 libhadooputils.a libhdfs.a libhdfs.so libhdfs.so.0.0.0

In fact, this problem is very simple to solve, I am here is the hadoop-2.6.0 version.

If you are also a hadoop2.6, you can download this:

Http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.6.0.tar

For other hadoop versions, download the following:

Http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.*.0.tar

[hadoop@djt002 native] $tar-xvf hadoop-native-64-2.6.0.tar-C $HADOOP_HOME/lib/native

[hadoop@djt002 native] $tar-xvf hadoop-native-64-2.6.0.tar-C $HADOOP_HOME/lib

Above, these two orders should be carried out.

Then add the environment variable

[root@djt002 native] # vim / etc/profile

Add the following:

Export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

Export HADOOP_OPTS= "- Djava.library.path=$HADOOP_HOME/lib"

Let the environment variable take effect

[root@djt002 native] # source / etc/profile

Result

Finally, the problem was solved successfully!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.