Shulou(Shulou.com)06/03 Report--

Big data cluster management is divided into manual mode (Apache hadoop) and tool mode (Ambari + hdp and Cloudera Manger + CDH).

Host:

192.168.0.32:Server+agent

192.168.0.33:agent

192.168.0.34:agent

Prepare the environment before installation:

Network name

IPv4 is required for CDH. IPv6 is not supported.

Configure each host in the cluster as follows to ensure that all members can communicate with each other:

Set the hostname to a unique name (not the local host).

Sudo hostnamectl set-hostname hadoop01.youe.com

The edit / etc / hosts file uses the IP address and fully qualified domain name (FQDN) of each host in the cluster. You can also add unqualified names.

192.168.0.32 hadoop01.youe.com

192.168.0.33 hadoop02.youe.com

192.168.0.34 hadoop03.youe.com

The canonical name / etc / hosts file for each host must be FQDN (for example, myhost-1.example.com), not an unqualified hostname (for example, myhost-1). The canonical name is the first entry after the IP address.

Do not use the alias / etc / hosts file or configure DNS.

An unqualified hostname (short name) must be unique within the Cloudera Manager instance. For example, you cannot have both host01.example.com and host01.standby.example.com managed by the same Cloudera Manager Server.

Edit / etc/sysconfig/network is limited to the FQDN of this host:

HOSTNAME = hadoop01.youe.com

Disable firewall and selinux (all sessions)

NTP (all sessions)

CDH requires that you configure the Network time Protocol (NTP) service on each computer in the cluster. To start NTP and configure it to run automatically on reboot, perform the following steps on each node in the cluster.

Yum-y install ntp

Edit / etc/ntp.conf file to add NTP server

Server 0.pool.ntp.org

Server 1.pool.ntp.org

Server 2.pool.ntp.org

Start ntp

Systemctl enable ntpd

Systemctl start ntpd

Synchronization node

Ntpdate-u 0.pool.ntp.org

Synchronize the system clock

Hwclock-systohc

Python version

Python2.7 (centos7 default python2.7)

Ssh key-free login: generate private and public keys (all nodes)

Ssh-keygen-t rsa-P''

Ssh-copy-id root@192.168.0.33

Ssh-copy-id root@192.168.0.34

Ssh-copy-id root@192.168.0.32

Master can connect to 32 33 34 through ssh

Install cloudera manager server

Install the repository (all nodes)

Wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo-P / etc/yum.repos.d/

Import the repository signature GPG key:

Rpm-- import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

Install jdk configuration environment variables (all sessions)

* The JDK must be 64-bit. Do not use a 32-bit JDK.

* The installed JDK must be a supported version as documented in Java Requirements.

* The same version of the Oracle JDK must be installed on each cluster host.

* The JDK must be installed at / usr/java/jdk-version.

After configuring the repository for Cloudera Manager, you can use the package manager to install Oracle JDK on the Cloudera Manager Server host, as follows:

Yum-y install oracle-j2sdk1.8

If you install jdk manually, you must install it to the / usr/java/ directory

Install the cloudera manager package

Master node:

Yum-y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server

Node node:

Yum-y install cloudera-manager-agent cloudera-manager-daemons

Cloudera-manager-daemons is a must for server and agent, and is about 1.3G in size.

You can also download the rpm package in advance to install: https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/

Wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm

Wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.0.0-530873.el7.x86_64.rpm

Wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm

If you install with the rpm package, install cloudera-manager-daemons first, then install cloudera-manager-server and cloudera-manager-agent

Enable Auto-TLS to automatically create certificates: (trouble configuring later, you can skip this item if https is not enabled)

Https://www.cloudera.com/documentation/enterprise/6/latest/topics/auto_tls.html#auto_tls

Note: Auto-TLS supports two options:

* option 1: use Cloudera Manager to generate internal certification authorities and corresponding certificates

* option 2: use an existing certification authority and the corresponding certificate

Option 1 is used here to enable auto-TLS to use the internal certification authority (CA) created and managed by Cloudera Manager. To use a trusted public CA (option 2), you must first obtain the certificate of the cluster host.

Auto-TLS must be enabled during the new installation. Cannot enable automatic TLS on an existing cluster. Auto-TLS must be enabled before any hosts can be added to Cloudera Manager.

After installing Cloudera Manager Agent on the Cloudera Manager Server host, before starting Cloudera Manager, enable auto-TLS as shown below.

JAVA_HOME=/usr/java/jdk1.8.0_141 / opt/cloudera/cm-agent/bin/certmanager setup-- configure-services

Replace the JDK version with the correct path for your environment. If you want to store files in a directory other than the default directory (/ var/lib/cloudera-scm-server/certmanager), add the-- location option as follows:

JAVA_HOME=/usr/java/jdk1.8.0_121 / opt/cloudera/cm-agent/bin/certmanager-location / opt/cloudera/CMCA setup-configure-services

/ opt/cloudera/CMCA this directory is created automatically and must not exist until the command is run. Check the / var/log/cloudera-scm-agent/certmanager.log log file to confirm that the directory has been created.

Install mysql:

Https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_mysql.html#cmig_topic_5_5_2

Wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

Yum-y install mysql-community-release-el7-5.noarch.rpm

Yum-y update

Yum-y install mysql-server

* for MySQL 5.6and 5.7, MySQL-shared-compat or MySQL-shared packages must be installed. This is required for the installation of the Cloudera Manager Agent package.

* if GTID-based replication is enabled in MySQL, the Cloudera Manager installation will fail.

1. Move old InnoDB log files / var/lib/mysql/ib_logfile0 and / var/lib/mysql/ib_logfile1 out of / var/lib/mysql/ to a backup location.

Rm-rf / var/lib/mysql/ib_logfile0 / var/lib/mysql/ib_logfile1

If you want to make changes to an existing database, be sure to stop using any services that use the database before continuing.

Vim / etc/my.cnf

# official recommended profile

[mysqld]

Datadir=/var/lib/mysql

Socket=/var/lib/mysql/mysql.sock

Transaction-isolation = READ-COMMITTED

Symbolic-links = 0

Key_buffer_size = 32m

Max_allowed_packet = 32m

Thread_stack = 256K

Thread_cache_size = 64

Query_cache_limit = 8m

Query_cache_size = 64m

Query_cache_type = 1

Max_connections = 5505

Log_bin=/var/lib/mysql/mysql_binary_log

Server_id=1

Binlog_format = mixed

Read_buffer_size = 2m

Read_rnd_buffer_size = 16m

Sort_buffer_size = 8m

Join_buffer_size = 8m

Innodb_file_per_table = 1

Innodb_flush_log_at_trx_commit = 2

Innodb_log_buffer_size = 64m

Innodb_buffer_pool_size = 4G

Innodb_thread_concurrency = 8

Innodb_flush_method = O_DIRECT

Innodb_log_file_size = 512m

[mysqld_safe]

Log-error=/var/log/mysqld.log

Pid-file=/var/run/mysqld/mysqld.pid

Sql_mode=STRICT_ALL_TABLES

Systemctl enable mysqld

Systemctl start mysqld

Initialize Mysql

/ usr/bin/mysql_secure_installation

[...]

Enter current password for root (enter for none): Asd@12345

OK, successfully used password, moving on...

[...]

Set root password? [Y/n] Y

New password:

Re-enter new password:

Remove anonymous users? [Y/n] Y

[...]

Disallow root login remotely? [Y/n] N

[...]

Remove test database and access to it [Y/n] Y

[...]

Reload privilege tables now? [Y/n] Y

All done!

Install the MySQL JDBC driver

Wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

Tar zxvf mysql-connector-java-5.1.46.tar.gz

Mkdir-p / usr/share/java/

Cd mysql-connector-java-5.1.46

Cp mysql-connector-java-5.1.46-bin.jar / usr/share/java/mysql-connector-java.jar

Create a database for Cloudera software: https://www.cloudera.com/documentation/enterprise/6/latest/topics/prepare_cm_database.html

The database must be configured to support MySQL UTF8 character set encoding.

Mysql-u root-pAsd@12345

Create database scm default character set utf8 default collate utf8_general_ci

Grant all on scm.* to 'scm'@'%' identified by' Admin@123'

Create database amon default character set utf8 default collate utf8_general_ci

Grant all on amon.* to 'amon'@'%' identified by' Admin@123'

Create database rman default character set utf8 default collate utf8_general_ci

Grant all on rman.* to 'rman'@'%' identified by' Admin@123'

Create database hue default character set utf8 default collate utf8_general_ci

Grant all on hue.* to 'hue'@'%' identified by' Admin@123'

Create database metastore default character set utf8 default collate utf8_general_ci

Grant all on metastore.* to 'hive'@'%' identified by' Admin@123'

Create database sentry default character set utf8 default collate utf8_general_ci

Grant all on sentry.* to 'sentry'@'%' identified by' Admin@123'

Create database nav default character set utf8 default collate utf8_general_ci

Grant all on nav.* to 'nav'@'%' identified by' Admin@123'

Create database oozie default character set utf8 default collate utf8_general_ci

Grant all on oozie.* to 'oozie'@'%' identified by' Admin@123'

Flush privileges

Set up the Cloudera Manager database

Mysql executes locally:

Rm-rf / etc/cloudera-scm-server/db.mgmt.properties

/ opt/cloudera/cm/schema/scm_prepare_database.sh

/ opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

/ etc/cloudera-scm-server/db.properties will be generated, which contains the connection information of the database. If this file is incorrect or does not exist, the later service cannot be started.

There may be an error: a compatible version of Java cannot be found on this host, either because javahome is not set or because a compatible version of Java is not installed.

Solution: the JAVA_HOME path is based on the path configured in / opt/cloudera/cm-agent/service/common/cloudera-config.sh.

Execute when mysql is not local:

Run the script (cm01.example.com) on the Cloudera Manager Server host and connect to the remote MySQL host (db01.example.com):

/ opt/cloudera/cm/schema/scm_prepare_database.sh mysql-h db01.example.com-- scm-host cm01.example.com scm scm

If you are installing CDH 6, do not select KAFKA,KUDU or SPARK packages because they are included in CDH 6

Download address manually:

Wget https://archive.cloudera.com/cdh7/6.0.0/parcels/CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel

Wget https://archive.cloudera.com/cdh7/6.0.0/parcels/CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha256

Wget https://archive.cloudera.com/cdh7/6.0.0/parcels/manifest.json

Place the CHD6-related Parcel package in the / opt/cloudera/parcel-repo/ directory of the primary node, or if you don't have it, you can create it yourself.

Note: finally, rename CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha256 to CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha, which must be noted, otherwise the system will download the CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel file again.

important! -- > in the manifest.json file, find the key of the corresponding version and copy it to the .sha file. If the key is wrong in the manifest.json file, the system will also download the CDH package again.

Chown cloudera-scm:cloudera-scm / opt/cloudera/-R

Chown cloudera-scm:cloudera-scm / var/log/cloudera-scm-agent-R

Disable transparent large page compression

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

And write the above two commands to boot self-startup.

Vim / etc/rc.local

Optimized switching partition: https://blog.csdn.net/lufeisan/article/details/53339991

Vim / etc/sysctl.conf

Vm.swappiness = 10

Sysctl-p / etc/sysctl.conf

Install CDH: https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_software_cm_wizard.html

After setting up the Cloudera Manager database, start Cloudera Manager Server and log in to Cloudera Manager Admin Console:

Start Cloudera Manager Server:

Systemctl start cloudera-scm-server

You can view the startup log

Tail-f / var/log/cloudera-scm-server/cloudera-scm-server.log

When you see this message, the startup is complete.

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

Log in

In the Web browser, go to 192.168.0.32 IP 7180, where is the FQDN or IP address of the host running Cloudera Manager Server.

If you enable automatic TLS, it is redirected to https://:7183 and a security warning is displayed. You may need to indicate that you trust the certificate, or click to continue with the Cloudera Manager Server host.

Log in to Cloudera Manager Admin Console. The default credentials are:

Username: admin

Password: admin

Error 1: could not contact scm server at localhost:7182, giving up during installation

BEGIN python-c 'import socket; import sys; s = socket.socket (socket.AF_INET); s.settimeout (5.0); s.connect ((sys.argv [1], int (sys.argv [2])); s.close ();' localhost 7182

Solution: delete the disconnected machine / usr/bin/host files so that Cloudera Manager will connect directly using ip.

Error 2:

Solution: look at the process, then kill it, and restart it.

Ps-ef | grep supervisord

Kill-9

Systemctl restart cloudera-scm-agent

If the deployment fails, you can uninstall and delete the Cloudera Manger environment and redeploy.

Systemctl stop cloudera-scm-server

Systemctl stop cloudera-scm-agent

Yum-y remove 'cloudera-manager-*'

Yum clean all

Umount cm_processes

Umount / var/run/cloudera-scm-agent/process

Rm-Rf / usr/share/cmf / var/lib/cloudera* / var/cache/yum/cloudera* / var/log/cloudera* / var/run/cloudera*

Rm-rf / tmp/.scm_prepare_node.lock

Rm-Rf / var/lib/flume-ng / var/lib/hadoop* / var/lib/hue / var/lib/navigator / var/lib/oozie / var/lib/solr / var/lib/sqoop* / var/lib/zookeeper

Rm-Rf data_drive_path/dfs data_drive_path/mapred data_drive_path/yarn

Systemctl stop mysqld

Yum-y remove mysql

Rm-rf / var/lib/mysql

Rm-rf / var/log/mysqld.log

Rm-rf / var/lib/mysql/mysql

Rm-rf / usr/lib64/mysql

Rm-rf / usr/share/mysql

Rm-rf / opt/cloudera/cm/cloudera-navigator-audit-server/schema/mysql

Rm-rf / opt/cloudera/cm/cloudera-navigator-server/schema/mysql

Rm-rf / opt/cloudera/cm/schema/mysql

Rpm-qa | grep-I mysql

Yum-y remove MySQL-python-1.2.5-1.el7.x86_64

Yum-y remove mysql-community-libs-5.7.23-1.el7.x86_64

Yum-y remove mysql57-community-release-el7-10.noarch

Yum-y remove mysql-community-common-5.7.23-1.el7.x86_64

Yum-y remove mysql-community-libs-compat-5.7.23-1.el7.x86_64

Yum-y remove mysql-connector-java-5.1.25-3.el7.noarch

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.