Shulou(Shulou.com)06/02 Report--

The online installation method of CDH is as follows, which is convenient, but because in China, the speed of visiting this website is very slow and not fast.

Install Cloudera Manager server online

# configure cloudera-cdh source and cloudera-manager source

# cloudera-manager

Wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

# cloudera-cdh

Wget https://archive.cloudera.com/cdh6/redhat/6/x86_64/cdh/cloudera-cdh6.repo

# install jdk and cloudera-manager

Sudo yum install cloudera-manager-daemons cloudera-manager-server-y

Then you can visit IP:7180 and operate through a web page. However, the reality is that online installation is very slow.

We use offline installation to do this:

I. basic environmental preparation

1. Download address of related packages:

Cloudera Manager address: http://archive.cloudera.com/cm5/cm/5/

To download the corresponding file, I downloaded: cloudera-manager-el6-cm5.13.3_x86_64.tar.gz

2. CDH installation package address: http://archive.cloudera.com/cdh6/parcels

Since our operating system is CentOS6.5, we need to download the following files and find the corresponding version:

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

Manifest.json

3. Prepare JDBC, which will be used to connect mysql later:

Http://download.softagency.net/MySQL/Downloads/Connector-J/

Updated on 20180727

The Mysql jar connection has been disabled. You can download the latest jar package at the following website.

Https://dev.mysql.com/downloads/file/?id=477054

4. All the required software has been downloaded, including mysql jar, there are a total of 5 files

Cloudera-manager-el6-cm5.13.3_x86_64.tar.gzCDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.jsomysql-connector-java-5.1.35-bin.jar

5. Prepare JDK

Tar-zxvf jdk1.7.0_79.tar.gz-C / usr/local/jvmcat / etc/profileexport JAVA_HOME=/usr/local/jvm/jdk1.7.0_79 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

6. Installation Settings NTP

6.1 yum install ntp

Cat / etc/ntp.confdriftfile / var/lib/ntp/drift restrict default kod nomodify notrap nopeer noqueryrestrict-6 default kod nomodify notrap nopeer noqueryrestrict 127.0.0.1 restrict-6:: 1 restrict 192.168.206.0 mask 255.255.255.0 nomodify notrap server 0.centos.pool.ntp.org iburst server 1.centos.pool.ntp.org iburst server 2.centos.pool.ntp.org iburst server 3.centos.pool.ntp.org iburst

6.2 start ntp:

$service ntpd start

6.3 set boot:

$chkconfig ntpd on

Check whether the setting is successful: chkconfig-- list ntpd where 2-5 indicates success if the on status is 2-5.

6.5 time synchronization can be performed first with the following command:

Ntpdate us.pool.ntp.org

6.6 execute on two slave nodes

Ntpdate-u bigdata1

7. Turn off firewall, selinux, etc.

Second, when the environment is ready, start installing CDH

It is recommended to use root to install CDH, and there are a lot of directories that need to be created by CDH. Users and other operations.

Decompress CM to the path of each node / opt

Tar-zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz-C / opt

Copy the downloaded mysql-connector-java.jar to each node

Cp mysql-connector-java-5.1.35-bin.jar / opt/cm-5.13.3/share/cmf/lib/scp mysql-connector-java-5.1.35-bin.jar root@bigdata2:/opt/cm-5.13.3/share/cmf/lib/scp mysql-connector-java-5.1.35-bin.jar root@bigdata3:/opt/cm-5.13.3/share/cmf/lib/

two。 Install mysql and prepare relevant users. Pay attention to give with grant option permission during the user creation process, otherwise an error will be reported later.

2.1 the yum source for installing MySQL, the following is the download address for the RHEL6 series mysql5.6:

Rpm-ivh http://repo.mysql.com/mysql-community-release-el6-5.noarch.rpm

2.2 yum installation

Yum install mysql-community-server-y

2.3 create users and empower

Grant all privileges on *. * to root@'localhost' identified by 'root' with grant option;grant all privileges on *. * to root@'%' identified by' root' with grant option;grant all privileges on *. * to scm@'localhost' identified by 'scm' with grant option;grant all privileges on *. * to scm@'%' identified by' scm' with grant option

3. Initialize the database for CM5 on the primary node:

/ opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql cm- hbigdata1-uroot-proot-- scm-host bigdata1 scm scm scm

Parameter explanation: database type, database name, database hostname, database username, password-- scm-host cmserver hostname scm scm scm

4. Agent configuration

Modify the hostname of the server_host primary node in / opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini. Otherwise agent won't be able to find the server side.

4.1Sync software to all Agent nodes:

Scp-r / opt/cm-5.13.3 root@node2:/opt/

5. Create cloudera-scm users on all nodes

Useradd-system-home=/opt/cm-5.13.3/run/cloudera-scm-server/-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

6. Prepare Parcels to install CDH5

Place the CHD5-related Parcel packages in the / opt/cloudera/parcel-repo/ directory of the primary node (parcel-repo needs to be created manually).

The relevant documents are as follows:

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

Manifest.jso

Finally, rename CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 to CDH-5.7.1- CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha, which must be noted, otherwise the system will download the CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 file again.

Updated 20180727

Originally this place is not easy to find, remove the "1" method of installation. Install now and find that the CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 name of "1" remains the same.

7. Related startup script

Start the server through / opt/cm-5.13.3/etc/init.d/cloudera-scm-server start.

Start the Agent service through / opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start.

(all nodes start Agent services, including servers)

What we are starting is actually a service script. We need to stop the service and change the above start parameters to stop. Restart is restart.

When you install it here, you can consider backing up the vmware virtual machine. If you encounter an error during the deployment of CM, you can roll to the current state adjustment.

III. Secure assembly and cluster configuration of CDH5

1. After Cloudera Manager Server and Agent are started, you can install and configure CDH5.

two。 At this point, you can access port 7180 of the main node through the browser to test (since it takes some time to start CM Server, it may take a while to access the page, perhaps 3-5 minutes). The default user name and password are admin.

3. After logging in, you will go to the Welcome / Authorization page

4. Select the authorized version, and select the "free version" here.

5. If the previous configuration is correct and the agent starts successfully, you can see all the nodes in the "currently managed hosts". Note: the firewall must be turned off, or there will be agent nodes that cannot be discovered.

6. The package has been placed in the directory / opt/cloudera/parcel-repo earlier, so you can see the local package directly here.

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha

Manifest.json

7. Click "continue" to the cluster installation, this will be faster, about 10 minutes to complete. If it is installed online, this will take several hours, or even the download fails.

8. As prompted, fix some items that have not been configured before, and then re-check. It is suggested that all errors should be solved to prevent weird problems in subsequent use.

9. Here, I have two problems:

9.1.JDK is not recognized. Make sure that the JDK path is recognized before running.

9.2. Perl package is required to be installed

Yum install-y perl

10. Select the combination that needs to be installed, or you can choose to customize it, and then install each component separately

11. Select each role on which machine you need to deploy

twelve。 Select the components that need to use mysql, the db, user name, password, etc., created in mysql.

13. After the installation is complete.

14. After the installation is complete, you can go to the cluster interface to take a look at the current state of the cluster.

There may be an error message that the query cannot be issued: the request for Service Monitor timed out. If there is no problem with the installation of each component, it is usually caused by the server comparing cards. Just refresh the page after a while, and adjust the configuration according to the actual situation:

15. In addition, if you want to add a new host, you must first start the agent service on the new host, otherwise the new host cannot be found, and it is troublesome to download cm-agent again. After you install the new node, remember to run the deploy client command.

IV. Summary of questions

Question 1

After the CDH environment is built, the following problems occur when visiting the Oozie management page:

Solution:

According to the hint, we can find that Ext JS library is missing, and we can click on the URL link given:

Oozie Quick Start

Find the following:

Then find the ExtJS2.2 download and upload it to the CDH server's directory / opt/cloudera/parcels/CDH/lib/oozie/libext/ after the download is complete.

The specific operations are as follows:

Cd / opt/cloudera/parcels/CDH/lib/oozie/libext

Unzip ext-2.2.zip

Chown oozie:oozie-R ext-2.2

Download ext-2.2.zip is no longer available on this official website. If you download it, please click the link:

Then we can visit the page again.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.