Shulou(Shulou.com)06/03 Report--

The method of virtual machine and Linux system installation has been described in detail in the previous two sharing articles, and each step has been illustrated. If there are friends who still don't understand, there's nothing I can do to help. This article focuses on the hadoop server operating system configuration in detail, hadoop installation will be described in detail in the next article.

The hadoop installation package uses a fast DKHadoop distribution. Personally, I think the installation process of DKHadoop is relatively simple. The installation of dkhadoop will be described in detail in the next sharing. Let's move on to the topic of this article-- Server operating system configuration tutorial.

First, prepare the installation package

1. Distributed installation of virtual machines (three or more virtual machines)

If it is a personal computer Windows system or Linux system, three virtual servers on the virtual machine, copy the installation package to the server and install it.

2. Physical cluster (three or more physical servers)

1. Download mode of public network

2. Local file mode

Copy the installation package file directly to the physical server root directory for installation.

3. Upload mode

This mode is applied when the server is in the computer room and there are no display and input devices.

Now install the files on the local computer (default laptop, on site in the computer room), connect the computer to the server, and upload the installation packages install and DKHInstall to the server root directory.

II. Server operating system configuration tutorial

After the preparation work is completed, the server configuration operation must be carried out. First of all, the three servers must communicate with each other by ping. The so-called ping connection means that the network between two devices is connected. When a data packet is sent from one end, it can be received by the other end, which means that the two devices can ping.

1. Modify permissions

Objective: to make the two installation packages of install and DKHInstall have executable permissions. Insufficient permissions to perform some operations. Install contains scripts and all the components, and DKHInstall contains the installation interface.

Step: modify the file permissions after copying the installation packages install and DKHInstall to the master node directory in the preparation work. First of all, enter the root directory, and the permissions of the installation directory install and DKHInstall are changed to: the file owner can read, write and execute, other users who belong to the same user group as the file owner can read and execute, and other user groups can read and execute.

Command:

Cd / root/

Unzip DKHPlantform.zip

Chmod-R 755 DKHPlantform

2. Set up Hadoop cluster and set up SSH secret-free login.

Purpose: the remote Hadoop daemon needs to be managed when Hadoop is running. After Hadoop is started, NameNode starts and stops various daemons on each DataNode through SSH (Secure Shell). This must be in the form of no password when executing instructions between nodes, so we need to configure SSH to use the form of password-free public key authentication, so that NameNode uses SSH to log in without password and start the DataName process. In the same principle, DataNode can also use SSH without password to log in to NameNode.

Steps:

(1) modify the local hosts file and write the corresponding relationship

In order to distinguish each host in the local area network, a host name is assigned to the host, and each host communicates through IP, but the IP address is not easy to remember, so configuring hostname and IP mapping can achieve fast and convenient access between hosts.

Command:

Vi / etc/hosts

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq, and then enter to save. Enter Q! After entering the car, you give up saving and exit.

After entering the editing mode, write the corresponding relationship between the host and the ip according to the rules (the host name dk41 is named by yourself, as shown below):

192.168.1.41 dk41

192.168.1.42 dk42

192.168.1.43 dk43

After editing, save and exit. Copy the correspondence to two or more other machines.

Command:

Scp-r / etc/hosts 192.168.1.42:/etc

Scp-r / etc/hosts 192.168.1.43:/etc

(2) perform pre-secret preparation between clusters.

When you execute the sshpass.sh script, you read the sshhosts and sshslaves files, replacing the master and slave files in sshpass.sh.

Modify the file sshhosts and enter the hostnames of all machines, one for each line (as shown below)

Command:

Vi / root/DKHPlantform/autossh/sshhosts

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq, and then enter to save. Enter Q! After entering the car, you give up saving and exit.

Modify the file sshslaves to write all machine names except the hostname (figure below)

Command:

Vi / root/DKHPlantform/autossh/sshslaves

Enter edit mode by pressing insert or I key on the keyboard, press Esc key after editing, then press Shift+: key, enter wq, and then enter to save. Enter Q! After entering the car, you give up saving and exit.

(3) perform cluster secret-free work

Command:

Cd / root/DKHPlantform/autossh

. / autossh host node hostname cluster password

Example:. / autossh dk41 123456

(4) turn off the firewall

To prevent some services from being blocked when accessing the server, you need to turn off the firewall.

Command:

Cd / root/DKHPlantform/autossh

. / offIptables.sh

3. Install MySQL with dual-computer hot backup

Objective: to store the metadata of Hive

Steps:

(1) distribute the mySQL installation directory from the primary node to the second node

Command:

Scp-r / root/DKHPlantform/mysqlInst/ 192.168.1.42:/root/

(2) the primary node executes:

Command:

Cd / root/DKHPlantform/mysqlInst/

. / mysql.sh 1

Execute from the node:

Command:

Cd / root/mysqlInst/

. / mysql.sh 2

(3) perform hot backup after successful execution (on both machines, the two ip are exchanged, 41 is written on 42, 42, 41, and the password is MySQL: 123456. It has been set in the platform, please do not modify it):

Command:

Source / etc/profile

. / sync.sh 192.168.1.xxx (another mysql address)

4. Create a database

Objective: MySQL is a kind of associated database management system, which stores data in different tables, which increases speed and flexibility.

Steps:

(1) Import the MySQL data table and execute it only on the master node:

Command:

Mysql-uroot-p123456 < {here is the sql file, and the file in the home directory: dkh.sql}

Such as: mysql-uroot-p123456

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.