Shulou(Shulou.com)06/03 Report--

Centos6 install cdh6.7

Pre-installation preparation JDK environment

Version: jdk-7u79-linux-x64.tar.gz

Download address: oracle official website (http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260-zhs.html)

/ usr/local/jdk1.7.0_79 will be generated on the installation

Need to establish a soft connection ln-s / usr/local/jdk1.7.0_79 / usr/local/java

One more ln-s / usr/local/java / usr/share/java

Also have a ln-s / usr/local/java / usr/java/jdk1.7

Add environmental variables

Export JAVA_HOME=/usr/local/java

Export PATH=$ {JAVA_HOME} / bin:$ {PATH}

Mysql connection

Http://dev.mysql.com/downloads/connector/j/

Mysql-connector-java-5.1.47-bin.jar, I downloaded this.

Put the downloaded package to / usr/local/

And rename mysql-connector-java.jar

CDH

Download cloudera manager installation package first. Download address: http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6- cm5.7.0_x86_64.tar.gz

Download CDH installation package (parcel package) download address: http://archive.cloudera.com/cdh6/parcels/5.7.0/

CDH-5.7.0-1.cdh6.7.0.p0.45-el6.parcel

CDH-5.7.0-1.cdh6.7.0.p0.45-el6.parcel.sha1

Manifest.json

Need these three above

What the system needs to prepare

Ntp time all machines should have the same time (Baidu Bar won't be simple)

Ssh mutual (root users)

Firewall selinux is not open (not under consideration)

Parsing Ip hostname of all servers should be included in Hosts

Echo 0 > / proc/sys/vm/swappiness

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Set it up and start it up.

Install cdh

-manager machine-operate cloudera-manager machine first

Mkdir-p / var/cloudera-scm-server

Mkdir-p / opt/cloudera/parcel-repo

Tar xf / opt/cloudera-manager-el6-cm5.7.0_x86_64.tar.gz-C / opt/cloudera-manager

Put the other package here / opt/cloudera/parcel-repo (make sure to replace the sha1 with this sha)

Useradd-system-home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

Chown cloudera-scm:cloudera-scm / var/cloudera-scm-server

Chown cloudera-scm:cloudera-scm / opt/cloudera-manager

Chown cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo

-manager machine

-all machines

Add user cloudera-scm to all machines

Useradd-system-home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

Vim / opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent/config.ini

# Hostname of the CM server.

Server_host=node2, the address of the machine to be pointed to manager, ip is also fine.

-all machines

-manager machine

The machine in manager points to cm server's database initialization

Method 1:

/ opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql-h2.1.1.1-uroot-pxxxx-- scm-host 2.2.2.2 scm scm scm

The above command requires root permission and no scm library to execute.

Method 2: (it is recommended that we have dba management for the database. People cannot give you permission for root to give you a library.)

Building database by create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Grant all privileges on scm.* to scm@'node2' identified by 'scm'; authorization

/ opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql-hnode1-- scm-host node2 scm scm scm # node1 is the address of the database Node2 is the manager address scm library scm user scm password

Vim / opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-server/db.properties to see what is generated

/ opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server start start the service

Modify / etc/init.d/cloudera-scm-server if startup fails

CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.0/etc/default

Add system services:

Cp / opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server / etc/init.d/cloudera-scm-server

Chkconfig-add cloudera-scm-server

Chkconfig cloudera-scm-server on

-manager machine

-all machines-

Similarly, set the cloudera-scm-agent under / opt/cloudera-manager/cm-5.7.0/etc/init.d/ to the system service and start it.

Cp cloudera-scm-agent / etc/init.d/

Chkconfig-add cloudera-scm-agent

If vim / etc/init.d/cloudera-scm-agent does not do so, it will report that the command in the startup command cannot find the path.

CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.0/etc/default

/ etc/init.d/cloudera-scm-agent start start

Copy / etc/init.d/cloudera-scm-agent to all machines

Vi / opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent/config.ini pay attention to this file to the machine that will be assigned to manager

Add system service and startup to each one.

Log in to cdh on the Web page to install other software

Default admin admin of machines accessing manager on port 7180

Here, pay attention to choosing a new machine. If you don't have the agent installed just now, it will be easier to install here than the one above.

Here is the choice of the version

At this point, a machine without agent installed

Select the current management host

Create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; remembers the authorization database

Grant all privileges on *. * to hive@'%' identified by 'xxx'

Just click and pass the test.

The Defaults requiretty in / etc/sudoers needs to be commented out.

In the installation service interface, when execution fails, click to view details, which will give a similar error message. View the related logs in the / opt/cloudera-manager/cm-5.4.1/run/cloudera-scm-agent/process directory. The above error message is found because of the JAVA_HOME problem. In the startup script, JAVA_HOME is not obtained based on environment variables, but is found in several directories it enumerates, so just pass your java path ln-s, such as my ln-s / usr/local/java / usr/java/jdk1.7.

If it still doesn't work, step back in the next step.

The installation of manager is completed at this point.

Add Service oozie

Create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci; database building authorization

Grant all privileges on *. * to oozie@'%' identified by 'xxxx'

Add kafka service

There's a red warning.

This role needs to activate the following additional Parcel before it can be started: [kafka].

The above return appears

It doesn't matter if you reported a mistake. Click on the cluster home page to modify the configuration and you don't need to complete the above one.

Modify the configuration and then start to solve all the records perfectly. This is the end of the installation. If you have any problems, please contact me to solve.

Replace kafka version 2

Https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_kafka, you can view the version correspondence.

Access the cdh dashboard:

Click the icon

Click configure again

Remote parcel address Click ➕ number to select

Https://archive.cloudera.com/kafka/parcels/2.0.0/ plus the parcel address of this version, click Save.

Click kafka in the cluster.

Stop the service.

Stop the corresponding version in the gift.

Click download to activate the new version when you find it.

/ opt/cloudera/parcels/ has added a soft connection of KAFKA

Execute a command to see what happens:

Kafka-topics-- zookeeper 10.10.32.21 describe ip is ZooKeeper

Look at the address of Leader: if there is no value of-1, it will be normal and there will be a problem.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.