Shulou(Shulou.com)06/03 Report--

In the last sharing article, I shared with you some of the installation preparations for running and deploying hadoop, and this one continues to share with you some personal learning experiences. I learned to use the fast release DKHadoop, so all the experience sharing is based on DKHadoop, which should be explained here first. Personally, I think DKHadoop is very friendly to beginners. Novice friends can quickly search the website to download a three-node dkhadoop to have a look.

This is the end of the gossip, this article focuses on the configuration of the server operating system, the writing may be a little long, need a little patience to read oh!

1. Modify permissions

Step: after copying the installation package DKHPlantform.zip to the master node / root/ directory in preparation, extract and modify the file permissions (the file owner is readable, writable and executable, other users belonging to a user group are readable and executable, and other user groups are readable and executable).

Command:

Cd / root/

Unzip DKHPlantform.zip

Chmod-R 755 DKHPlantform

two。 Set up hadoop cluster and set up SSH secret-free login

Steps:

Modify hostname vi / etc/sysconfig/network reboot

(1) modify the local hosts file and write the corresponding relationship

Command:

Vi / etc/hosts

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq and enter to save. Enter Q! After entering the car, you give up saving and exit.

After entering the editing mode, write the corresponding relationship between the host and the ip according to the rules (the host name dk41 is named by yourself, as shown below):

192.168.1.41 dk41

192.168.1.42 dk42

192.168.1.43 dk43

After editing, save and exit. Copy the correspondence to the other two machines.

Command:

Scp-r / etc/hosts 192.168.1.42:/etc

Scp-r / etc/hosts 192.168.1.43:/etc

(2) perform pre-secret preparation between clusters.

A) when you execute the sshpass.sh script, you will read the sshhosts and sshslaves files, replacing the master and slave files in sshpass.sh.

Modify the file sshhosts and enter the hostnames of all machines, one for each line (as shown below)

Command:

Vi / root/DKHPlantform/autossh/sshhosts

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq, and then enter to save. Enter Q! After entering the car, you give up saving and exit.

Modify the file sshslaves to write all machine names except the hostname (figure below)

Command:

Vi / root/DKHPlantform/autossh/sshslaves

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq, and then enter to save. Enter Q! After entering the car, you give up saving and exit.

B) execute insExpect.sh, the system will install two rpm packages, no need to enter password and yes during the execution of sshpass.sh, and the script will continue to execute automatically.

Command:

Cd / root/DKHPlantform/autossh

. / insExpect.sh

Follow the prompts to enter the yes and password multiple times (as shown below)

C) execute the changeMaster.sh script in order to empty all files in the / root/.ssh directory (in the / root/DKHPlantform/autossh directory) and avoid conflicts with the old keys when the new keys are generated during the execution of the sshpass.sh scripts.

Command:

. / changeMaster.sh

Follow the prompts to enter (as shown below)

(3) perform cluster confidentiality (SSH)

A) execute SSH:

Command:

Cd / root/

. / sshpass.sh password

123456 is the cluster password, enter it according to your actual situation.

B) to prevent some services from being blocked when accessing the server, the firewall needs to be turned off.

Command:

Cd / root/DKHPlantform/autossh

. / offIptables.sh

3. Install MySQL with dual hot backup

Objective: to store the metadata of Hive

Steps:

(1) distribute the mysql installation directory from the primary node to the second node

Command:

Scp-r / root/DKHPlantform/mysqlInst/ 192.168.1.42:/root/

(2) the primary node executes:

Command:

Cd / root/DKHPlantform/mysqlInst/

. / mysql.sh 1

Ssh goes to the second machine (slave node) and executes:

Command:

Cd / root/mysqlInst/

. / mysql.sh 2

(3) perform hot backup after successful execution (on both machines, the two ip are exchanged, 41 is written on 42, 42, 41, and the password is Mysql: 123456. It has been set in the platform, please do not modify it):

Command:

Source / etc/profile

. / sync.sh 192.168.1.xxx (another mysql address) root 123456

4. Create a database

Steps:

(1) Import the MySQL data table and execute it only on the master node:

Command:

Mysql-uroot-p123456 < {here is the sql file, and the file in the home directory: dkh.sql}

Such as: mysql-uroot-p123456

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.