Shulou(Shulou.com)06/02 Report--

Detailed documentation of CDH cluster building steps

First, about CDH and Cloudera Manager

CDH (Cloudera's Distribution,including Apache Hadoop), one of the many branches of Hadoop, is maintained by Cloudera, built on a stable version of Apache Hadoop, and integrates many patches that can be used directly in production environments.

Cloudera Manager is to facilitate the installation, monitoring and management of related services such as Hadoop in the cluster, which greatly simplifies the installation and configuration management of hosts, Hadoop, Hive, Spark and other services in the cluster.

II. System environment

CDH Cluster Server Planning: 192.168.2.41 (CDH1), 192.168.2.51 (CDH2), 192.168.2.61 (CDH3)

Operating system: CentOS6.5 x64

Cloudera Manager:5.8.0

CDH: 5.8.0

Third, installation instructions

Using parcel offline installation mode, installation is convenient, reinstallation and so on are very convenient. The later upgrade of the cluster unified package is also very good.

4. Download address of related packages

Download address of Cloudera Manager:

Http://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.8.0_x86_64.tar.gz

CDH installation package address:

Http://archive.cloudera.com/cdh6/parcels/5.8.0/

Since our operating system is CentOS6.5, we need to download the following files:

CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel

CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel.sha1

Manifest.json

Fifth, installation preparation: system environment configuration

The following actions are performed by root users.

1. Network configuration (all nodes)

Vi / etc/sysconfig/network modify hostname:

It takes effect to restart the network service through service network restart.

Vi / etc/hosts, modify the correspondence between ip and hostname

Note: here you need to write in the ip and hostname correspondence of each machine, as well as those of the local machine, otherwise you will prompt hostname parsing error when starting Agent.

2. Configure SSH password-free login

Execute ssh-keygen-t rsa all the way on the primary node to generate a key pair without a password.

Then execute the following command to copy the public key file to all datenode nodes:

[root@cdh2 ~] # ssh-copy-id cdh2 [root@cdh2 ~] # ssh-copy-id cdh3 [root@cdh2 ~] # ssh-copy-id cdh4

Test: ssh cdh3 on the master node. Normally, you can log in without a password.

3. Install Java of Oracle (all nodes)

CentOS comes with OpenJdk, but running CDH5 requires Oracle's Jdk and Java 7 support.

Uninstall the OpenJdk that comes with it, use rpm-qa | grep java to query java-related packages, and uninstall it using rpm-e-- nodeps package name.

Go to Oracle's website to download jdk.

Http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz

Extract the installation package

[root@cdh2~] # tar-zxf jdk-7u79-linux-x64.tar.gz-C / usr/local [root@cdh2~] # cd/usr/local [root@cdh2~] # ln-sv jdk1.7.0_79/jdk [root@cdh2~] # vi / etc/profile.d/jdk.sh exportJAVA_HOME=/usr/local/jdkexportPATH=$PATH:$JAVA_HOME/bin [root@cdh2~] # chmod 755 / etc/profile.d/jdk.sh [root@cdh2~] # / etc/profile.d/jdk.sh

Check to see if the installation is successful

[root@cdh2~] # java-versionjavaversion "1.7.0,79" Java (TM) SE Runtime Environment (build 1.7.0_79-b15) JavaHotSpot (TM) 64-Bit Server VM (build 24.79-b02, mixed mode) [root@cdh2~] #

4. Install and configure MySQL (master node)

Install the mysql server through yum installmysql-server. Chkconfig mysqld on is set to boot, and service mysqld start starts the mysql service.

Mysql-uroot-pxxxx goes to the mysql command line and creates the following databases:

Createdatabase hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;createdatabase hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;createdatabase Reports DEFAULT CHARSET utf8 COLLATE utf8_general_ci;createdatabase Oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci

Set root to authorize access to all of the above databases:

# authorized root users have access to all databases on the master node

Grant all privileges on *. * to root@cdh2 identified by '123456investors Fushprivileges

5. Turn off firewall and SELinux (all nodes)

Note: it needs to be executed on all nodes because there are too many ports involved. The temporary shutdown of the firewall is more convenient to install. After installation, you can set the firewall policy as needed to ensure cluster security.

Turn off the firewall:

Service iptables stop (temporary shutdown) chkconfig iptables off (effective after restart)

Close SELinux

Setenforce 0 (temporary) modify the SELINUX=disabled under / etc/selinux/config (permanent after restart)

6. Synchronization time

Each cluster node server must synchronize the time. If the time difference is too big, a variety of abnormal errors may occur in the installation of CDH.

Install ntp

Yum install ntp ntpdate

Ntp synchronization time command. 1.cn.pool.ntp.org is used as the time synchronization server here.

Ntpdate 1.cn.pool.ntp.org VI, installation and startup 1, installation of Cloudera Manager Server and Agent

Decompression and installation of master node

The default location of the cloudera manager directory is / opt. Extract it to the opt directory.

[root@cdh2opt] # tar-zxvf cloudera-manager-el6-cm5.8.0_x86_64.tar.gz-C / opt

Change the owner of the extracted two directories to root

Chown-R root.root / opt/cloudera/chown-R root.root / opt/cm-5.8.0/

To build a database for Cloudera Manager 5, you first need to go to MySql's official website to download the JDBC driver.

Address: http://dev.mysql.com/downloads/connector/j/

After decompressing, find mysql-connector-java-5.1.41-bin.jar and put it in / opt/cm-5.8.0/share/cmf/lib/.

Cp mysql-connector-java-5.1.41-bin.jar / opt/cm-5.8.0/share/cmf/lib/

Initialize the database for CM5 on the primary node:

/ opt/cm-5.8.0/share/cmf/schema/scm_prepare_database.shmysql cm-h localhost-uroot-p123456-scm-host localhost scm scm scm

Agent configuration:

Modify the hostname of the server_host primary node in / opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini.

Server_host=cdh2

Create a pid directory for cloudera-scm-agent

Mkdir / opt/cm-5.8.0/run/cloudera-scm-agent

Synchronize Agent to other nodes

Scp-r/opt/cm-5.8.0/ cdh3:/opt/scp-r/opt/cm-5.8.0/ cdh4:/opt/

Create cloudera-scm users on all nodes

Useradd-system-home=/opt/cm-5.8.0/run/cloudera-scm-server/-no-create-home--shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

2. Prepare Parcels to install CDH5

Place the CHD5-related Parcel packages in the / opt/cloudera/parcel-repo/ directory of the primary node.

The relevant documents are as follows:

CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel

CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel.sha1

Manifest.json

Finally, change the suffix of CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel.sha1 file to sha. Otherwise, the system downloads the CDH-5.8.0-1.cdh6.8.0.p0.42-el6.parcel file again.

3. Start the service

Server:

/ opt/cm-5.8.0/etc/init.d/cloudera-scm-serverstart

Client: (all nodes start) start the Agent service.

/ opt/cm-5.8.0/etc/init.d/cloudera-scm-agentstart

What we are starting is actually a service script. We need to stop the service and change the above start parameters to stop. Restart is restart.

4. Installation and configuration of CDH5

Once Cloudera Manager Server and Agent are started, you are ready to install and configure CDH5.

At this point, you can access port 7180 of the main node through the browser to test (since it takes some time to start CM Server, it may take a while to access it here). The default user name and password are admin.

Check the acceptance terms and click continue

As you can see, the free version of CM5 no longer has the original limit of 50 nodes.

Click to continue

After each Agent node starts normally, you can see the corresponding node in the list of hosts currently managed. Select the node to install and click to continue.

The following package name and the corresponding CDH version appear, indicating that the local Parcel package is configured correctly. Select 5.8.0 here and click to continue.

If the local Parcel package is configured correctly, the download in the following figure should be completed in an instant, and then just wait patiently for the allocation process, which takes more than 10 minutes, depending on the speed of the private network.

Next is the server check, which may encounter the following problems:

Cloudera recommends that / proc/sys/vm/swappiness be set to 0. Currently set to 60. Use the sysctl command to change the setting at run time and edit / etc/sysctl.conf to save the setting after restart. You can continue with the installation, but you may encounter problems, and Cloudera Manager reports that your host is not performing well due to swapping. The following hosts are affected:

It can be solved by echo 0 > / proc/sys/vm/swappiness. Click finish.

Next, choose to install the service.

Assign roles to cluster settings. If there are no special requirements, you can choose the default and click to continue.

Database Settin

Click on the connection test, when all show Successful, click to continue

Select the default and click to continue.

Start installing the configuration service and wait for it to complete.

Installation complete, log in to the console

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.