Shulou(Shulou.com)06/02 Report--

This article mainly explains "the method of building big data cluster by cm+cdh". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "the method of building big data cluster by cm+cdh".

Part one: setting up the basic environment

1. Network configuration

Vim / etc/sysconfig/network-scripts/ifcfg-ens32

Service network restart

Vim / etc/hosts

192.168.15.121 node1

192.168.15.122 node2

192.168.15.123 node3

2. SSH key-free login

Ssh-keygen-t rsa-P''- f ~ / .ssh/id_rsa

Ssh-copy-id node1

Ssh-copy-id node2

Ssh-copy-id node3

Cd / .ssh

Scp. / authorized_keys node2: `pwd`

Scp. / authorized_keys node3: `pwd`

3. Firewall is off

Systemctl stop firewalld.service

Systemctl disable firewalld.service

4. SELINUX is closed

Setenforce 0

Vim / etc/selinux/config (SELINUX=disabled)

5. Install JDK configuration environment variables

Mkdir / usr/java

Download the rpm file of jdk on the official website and upload it to the / usr/java directory (e.g. / usr/java/jdk-8u221-linux-x64.rpm)

Scp. / jdk-8u221-linux-x64.rpm node2: `pwd`

Scp. / jdk-8u221-linux-x64.rpm node3: `pwd`

Rpm-ivh jdk-8u221-linux-x64.rpm

Vim / etc/profile

Export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

Export PATH=$JAVA_HOME/bin:$PATH

Export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Source / etc/profile

Java-version

6. Install NTP and set boot (if the time is the same, you can not synchronize the time, you can bypass this step)

Chkconfig ntpd on sets time synchronization ntpdate 202.120.2.101

7. Installation and configuration

Yum install-y mysql-server

Systemctl start mysqld.service # launch mysql

Systemctl restart mysqld.service # restart mysql

Systemctl stop mysqld.service # stop mysql

Systemctl enable mysqld.service # set mysql to boot

Mysql

Use mysql

Delete from user

Update user set Password=password ("admin") where User='root'

GRANT ALL PRIVILEGES ON *. * TO 'root'@'%' IDENTIFIED BY' admin' WITH GRANT OPTION

Flush privileges

8. Download the third-party dependency package

Yum-y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb

The second part: build cm+cdh

1. ClouderaManager installation

1.1. install Cloudera Manager Server, Agent

Mkdir / opt/cloudera-manager

Tar-zxf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz-C / opt/cloudera-manager/

1.2.Configuring CM Agent

Modify server_host to node1 in file vim / opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.3.Create user cloudera-scm

Useradd-system-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

1.4.Configuring CM Server database (node1)

Mkdir / usr/share/java/

Copy the mysql jar file to the directory / usr/share/java/

Note that the jar package name should be changed to mysql-connector-java.jar

Grant all on *. * to 'temp'@'%' identified by' temp' with grant option

Cd / opt/cloudera-manager/cm-5.16.1/share/cmf/schema/

. / scm_prepare_database.sh mysql temp-h node1-utemp-ptemp-- scm-host node1 scm scm scm

Format: database type, database, database server, user name, password, cm server server

1.5.Create Parcel directory

Mkdir-p / opt/cloudera/parcel-repo

Chown cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo

Mkdir-p / opt/cloudera/parcels

Chown cloudera-scm:cloudera-scm / opt/cloudera/parcels

1.6.Create CDH local source

Download the files CDH-5.4.0-1.cdh6.4.0.p0.27-el6.parcel and manifest.json, and place them under / opt/cloudera/parcel-repo on the server node.

Open the manifest.json file, which contains the configuration in json format, find the hash code corresponding to the download version, create a new file with the same file name as your parel package, and add the .sha suffix to copy the hash code to the file to save.

1.7. start CM Server, Agent

Cd / opt/cloudera-manager/cm-5.16.1/etc/init.d/

. / cloudera-scm-server start Sever

Starting for the first time will automatically create tables and data. Do not shut down or restart immediately, or you will need to delete all tables and data and reinstall them.

Check to see if startup is complete

Cd / opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server

Tail-f cloudera-scm-server.log

After the startup is complete

Visit: http://note1 's ip:7180

Default username and password: admin/admin

The installation was successful.

At this point, I believe you have a deeper understanding of "cm+cdh 's method of building big data cluster". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.