Shulou(Shulou.com)06/01 Report--

This article mainly explains the "RedHat hadoop cluster installation steps", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "RedHat hadoop cluster installation steps" bar!

1. Network configuration 2, machine name modification 3, cluster hosts list

4. Turn off the firewall and SELinux

# View selinux status

[root@master1] # sestatus-v

# turn off selinux

Vi / etc/selinux/config

SELINUX=disabled

5. Time synchronization # manually synchronize time

/ usr/sbin/ntpdate cn.pool.ntp.org

# configure time synchronization (to achieve automatic synchronization)

0 1 * / usr/sbin/ntpdate cn.pool.ntp.org

Crontab-e

6. Jdk environment installation

# View the jdk installed automatically by the system

[root@master1 opt] # rpm-qa | grep jdk

# remove jdk installed automatically by the system

[root@master1 opt] # rpm-qa | grep jdk

Java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

Java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

[root@master1 opt] # rpm-e-nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

-nodeps: unknown option

[root@master1 opt] # rpm-e-- nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

[root@node1 soft-src] # pwd

/ tmp/soft-src

# View contents of the directory

[root@node1 soft-src] # ll-a

The total dosage is 139052

Drwxr-xr-x. 3 root root 4096 September 22 10:30.

Drwxrwxrwt. 25 root root 4096 September 22 10:30..

-rwxrw-rw-. 1 root root 142376665 September 20 11:16 jdk-7u67-linux-x64.tar.gz

# decompression

[root@node1 soft-src] # tar xvf jdk-7u67-linux-x64.tar.gz

# View catalog

[root@node1 soft-src] # ll-a

The total dosage is 139052

Drwxr-xr-x. 3 root root 4096 September 22 10:30.

Drwxrwxrwt. 25 root root 4096 September 22 10:30..

Drwxr-xr-x. 8 uucp 143 4096 July 26 00:51 jdk1.7.0_67

-rwxrw-rw-. 1 root root 142376665 September 20 11:16 jdk-7u67-linux-x64.tar.gz

# set JAVA_HOME

[root@secondarynamenode Desktop] # vi / etc/profile.d/java.sh

Export JAVA_HOME=/tmp/soft-src/jdk1.7.0_67

Export CLASSPATH=/tmp/soft-src/jdk1.7.0_67/lib

Export PATH=$JAVA_HOME/bin:$PATH

# View the location where jdk is installed

[root@secondarynamenode Desktop] # which java

/ tmp/soft-src/jdk1.7.0_67/bin/java

7. Create hadoop users and hadoop groups

Groupadd hadoop

Useradd hadoop-g hadoop

# modify wqj user password after login by administrator

Passwd wqj

8. Generate ssh key

For configuration details, see "RedHat hadoop ssh configuration installation"

9. Create / opt/module/hadoop and / opt/data/hadoop directories 10, modify / opt/module/hadoop and / opt/data/hadoop directory permissions

Example chmod-R 700path

Mkdir-p path

11. Configure hadoop environment to decompress

[root@node1 modules] # tar xvf hadoop-1.0.3.tar

# modify the permissions of hadooop-1.0.3.tar

Chown hadoop:hadoop / opt/module/hadoop-1.0.3.tar

# configure HADOOP_HOME1), $HADOOP_HOME/conf/hadoop-env.sh

Export JAVA_HOME=/tmp/soft-src/jdk1.7.0_67

Export HADOOP_HEAPSIZE=512

2), / etc/profile.d/hadoop.sh

Export HADOOP_HOME=/opt/modules/hadoop-1.0.3

Export CLASSPATH=.:$HADOOP_HOME/lib:$CLASSPATH

Export PATH=.:$HADOOP_HOME/bin:$PATH

12. Configure the cluster

1), configure core-site.xml

This configuration file defines the configuration items for Hadoop Core, such as HDFS and MapReduce's commonly used Icano settings, etc.

Fs.default.name hdfs://master:9000 true

2), configure hdfs-site.xml

Dfs.name.dir / opt/data/hadoop/hdfs/name,/data1/hadoop/hdfs/name true dfs.data.dir / opt/data/hadoop/hdfs/data,/data1/hadoop/hdfs/data true fs.checkpoint.dir/opt/data/hadoop/hdfs/secondarynamenode / data1/hadoop/hdfs/secondarynamenode true dfs.http.address master:50070 true dfs.secondary.http.address 192.168.232.146:50090 true dfs.permissions false true dfs.hosts / etc/hadoop/conf/include true

3), configure mapred-site.xml

Mapred.job.tracker master:9001 true mapred.local.dir/opt/data/hadoop/mapred/local / data1/hadoop/mapred/local true mapred.system.dir / tmp/hadoop/mapred/system true mapred.tasktracker.map.tasks.maximum 7 true mapred.tasktracker.reduce.tasks.maximum 7 true mapred.child.java .opt-Xmx400m true mapred.hosts / etc/hadoop/conf/include true mapred.jobtracker.restart.recover true

4), format hdfs

Hadoop namenode-format

5) modify / opt/modules/hadoop directory permission to hadoop

6), modify / opt/data/hadoop/hdfs/name and other files with permissions of hadoop

7),

# start master node:

Hadoop-daemon.sh start namenode

# start data node:

Hadoop-daemon.sh start datanode

# start jobtracker

Hadoop-daemon.sh start jobtracker

# start tasktracker

Hadoop-daemon.sh start tasktracker

8),

View log information

[root@master Desktop] # tail-100f / opt/modules/hadoop-1.0.3/libexec/../logs/hadoop-root-datanode-master.log

Thank you for reading, the above is the content of "installation steps of RedHat hadoop cluster". After the study of this article, I believe you have a deeper understanding of the installation steps of RedHat hadoop cluster, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.