Shulou(Shulou.com)06/02 Report--

This article mainly explains "the installation process of Cloudera Manager and CDH 6.0.1 under CentOS7". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the installation process of Cloudera Manager and CDH 6.0.1 under CentOS7".

Concept introduction 1. Overview of CDH

CDH is the most complete, tested, and popular distribution of Apache Hadoop and related projects. CDH provides the core elements of Hadoop-scalable storage and distributed computing-as well as a web-based user interface and important enterprise functions. CDH is open source for Apache licensing and is the only Hadoop solution that provides unified batch processing, interactive SQL and interactive search, and role-based access control.

CDH provides the following features:

Flexibility: store any type of data and operate using a variety of different computing frameworks, including batch processing, interactive SQL, free text search, machine learning, and statistical computing.

Integration: get up and running quickly on a complete Hadoop platform that works with a wide range of hardware and software solutions.

Security: processing and controlling sensitive data.

Scalability: enable a wide range of applications and sizes and expand them to meet your needs.

High availability: perform mission-critical business tasks with confidence.

Compatibility: leverage your existing IT infrastructure and investment.

The CDH components are shown below:

2. Overview of Cloudera Manager

Cloudera Manager is an end-to-end application for managing cdh clusters. Cloudera Manager sets enterprise deployment standards by providing fine-grained visibility and control over each part of the CDH cluster-empowering operators to improve performance, improve quality of service, improve compliance, and reduce management costs. With Cloudera Manager, you can easily deploy and centralize the complete CDH stack and other managed services. The application automates the installation process, reducing deployment time from weeks to minutes, providing you with a real-time view of running hosts and services across the cluster, and providing a single central console to perform configuration changes across the cluster. and integrate various reporting and diagnostic tools to help you operate.

The architecture of Cloudera Manager is shown in the figure above (cs structure), which is mainly composed of the following parts:

Server / Server:

The core of Cloudera Manager. It is mainly used to manage web server and application logic. It is used to install software, configure, start and stop services, and manage clusters where services are running.

Agent / agent:

Installed on each host. It is responsible for starting and stopping processes, deploying configurations, triggering installation and monitoring hosts.

Database / Database:

Store configuration and monitoring information. Multiple logical databases that can typically run on one or more database servers. For example, the Cloudera manager service and monitoring, the daemon uses different logical databases.

Cloudera Repository: a software distribution library provided by cloudera manager.

Client / Clients:

Provides an interface to interact with Server.

The main role of Cloudera Manager, including server and agent;server, is to monitor cluster distribution, configure cluster, etc., and the server is in charge of each node of the cluster.

CDH is the installation package for CM, either locally or in the cloud, including all the components required by hadoop's ecosystem, which are managed and installed centrally through Cloudera Manager.

CDH can be installed not only through cm but also through yum,tar,rpm.

2. Environment preparation 1. Software version selection category version download address operating system CentOS Linux release 7.5.1804 (Core) use Ali Cloud Image download database 5.5.60-MariaDByum source installation JDKjdk-8u162-linux-x64.rpmOracle official website download Cloudera ManagerCloudera Manager 6.0.1 official warehouse CDHCDH 6.0.1 use parcels installation 2, node preparation (four nodes) name IPCM management software namenode01.wzlinux.com172.18.8.200Cloudera Manager Server&Agent MariaDBdatanode01.wzlinux.com172.18.8.201Cloudera Manager Agentdatanode02.wzlinux.com172.18.8.202Cloudera Manager Agentdatanode03.wzlinux.com172.18.8.203Cloudera Manager Agent

3. Configure hostname and hosts resolution (all nodes)

Edit / etc/hostname, modify the hostname, and use the command hostname to make it take effect immediately.

Edit the file / etc/hosts and add the following.

172.18.8.200 namenode01.wzlinux.com namenode01172.18.8.201 datanode01.wzlinux.com datanode01172.18.8.202 datanode02.wzlinux.com datanode02172.18.8.203 datanode03.wzlinux.com datanode034, turn off the firewall systemctl stop firewalld.servicesystemctl disable firewalld.service5, close SELinuxsed-I's closed SELINUXForcingSELINUX disabled ledSecretg'/ etc/selinux/configsetenforce 06, Add scheduled task echo "$((RANDOM%60)) $((RANDOM%24)) * / usr/sbin/ntpdate time1.aliyun.com" > > / var/spool/cron/root7, disable transparent large page compression

CDH configuration requires

Echo never > / sys/kernel/mm/transparent_hugepage/defragecho never > / sys/kernel/mm/transparent_hugepage/enabled

And write the above two commands to boot / etc/rc.local.

8. Optimize echo "vm.swappiness = 10" > > / etc/sysctl.confsysctl-p III, install CM and CDH1, configure Cloudera Manager warehouse (all nodes) wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo-P / etc/yum.repos.d/rpm-- import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

It is slow to install using the warehouse. It is recommended to download the required rpm for offline installation or build a private warehouse. There are three main software packages:

Cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpmcloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmcloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

Cloudera-manager-daemons is a must for server and agent.

2. Configure JDK (all nodes) rpm-ivh jdk-8u162-linux-x64.rpm3, install CM Server and Agent

It is recommended to install offline, download the rpm package to the server, send a copy to other nodes, and then install it locally, it will be much faster.

Namenode01:

Yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

Datanode0 [1-3]:

Yum install cloudera-manager-daemons cloudera-manager-agent4, installation database (namenode01)

The version of MariaDB that comes with CentOS 7.5 is supported.

Yum install mariadb-server-y

Edit the configuration file / etc/my.cnf and change it to the following:

[mysqld] datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.socktransaction-isolation = READ-COMMITTED# Disabling symbolic-links is recommended to prevent assorted security risks # to do so, uncomment this line:symbolic-links = Settings user and group are ignored when systemd is used.# If you need to run mysqld under a different user or group # customize your systemd unit file for mariadb according to the# instructions in http://fedoraproject.org/wiki/Systemdkey_buffer = 16Mkey_buffer_size = 32Mmax_allowed_packet = 32Mthread_stack = 256Kthread_cache_size = 64query_cache_limit = 8Mquery_cache_size = 64Mquery_cache_type = 1max_connections = 550#expire_logs_days = 10#max_binlog_size = 100M#log_bin should be on a disk with enough free space.#Replace'/ var/lib/mysql/mysql_binary_log' with An appropriate path for your#system and chown the specified folder to the mysql user.log_bin=/var/lib/mysql/mysql_binary_log#In later versions of MariaDB If you enable the binary log and do not set#a server_id, MariaDB will not start. The server_id must be unique within#the replicating group.server_id=1binlog_format = mixedread_buffer_size = 2Mread_rnd_buffer_size = 16Msort_buffer_size = 8Mjoin_buffer_size = 8M# InnoDB settingsinnodb_file_per_table = 1innodb_flush_log_at_trx_commit = 2innodb_log_buffer_size = 64Minnodb_buffer_pool_size = 4Ginnodb_thread_concurrency = 8innodb_flush_method = O_DIRECTinnodb_log_file_size = 512m [mysqld_safe] log-error=/var/log/ Mariadb/mariadb.logpid-file=/var/run/mariadb/mariadb.pid## include all files from the config directory#!includedir / etc/my.cnf.d

Open the database

Systemctl enable mariadbsystemctl start mariadb

Initialize the database

/ usr/bin/mysql_secure_installation

Follow the prompts below to enter.

[...] Enter current password for root (enter for none): OK, successfully used password, moving on... [...] Set root password? [Y/n] YNew password:Re-enter new password: [...] Remove anonymous users? [Y/n] Y [...] Disallow root login remotely? [Y/n] N [...] Remove test database and access to it [Y/n] Y [...] Reload privilege tables now? [Y/n] Y [...] All done! If you've completed all of the above steps, your MariaDBinstallation should now be secure.Thanks for using MariaDBMS 4, install MySQL JDBC driver (all nodes)

Used for each node to connect to the database.

Wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gztar xf mysql-connector-java-5.1.46.tar.gzmkdir-p / usr/share/java/cd mysql-connector-java-5.1.46cp mysql-connector-java-5.1.46-bin.jar / usr/share/java/mysql-connector-java.jar5, create a database for each Cloudera software

Log in to the database using root and create the following database and account.

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY' wzlinux' CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY' wzlinux';CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY' wzlinux' 6 、 Set up Cloudera Manager database [root@namenode01 ~] # / opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scmEnter SCM password: JAVA_HOME=/usr/java/jdk1.8.0_162Verifying that we can write to / etc/cloudera-scm-serverCreating SCM configuration file in/ etc/cloudera-scm-serverExecuting: / usr/java/jdk1.8.0_162/bin/java-cp / usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle -connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor / etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [main] DbCommandExecutor INFO Successfully connected to database.All done Your SCM database is configured configuration 7, install the CDH configuration CDH package parcels (namenode01)

After the CM installation is successful, we can build the enterprise big data platform by installing CDH through CM. So first you need to download the parcels package of CDH to the CM master server.

Similarly, in order to speed up our installation, we can download the packages that need to be downloaded in advance, or we can create a private CDH repository.

[root@namenode01 ~] # cd / opt/cloudera/parcel-repowget https://archive.cloudera.com/cdh7/6.0.1/parcels/CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcelwget https://archive.cloudera.com/cdh7/6.0.1/parcels/manifest.json

In the manifest.json file, find the key of the corresponding version and copy it to the .sha file.

Echo "2e650f1f1ea020a3efc98a231b85c2df1a50b030" > CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel.sha

Modify to belong to the master group.

Chown cloudera-scm.cloudera-scm / opt/cloudera/parcel-repo/* starts Cloudera Manager Server [root@namenode01 ~] # systemctl start cloudera-scm-server

If there are any problems during startup, you can check the log.

Tail-f / var/log/cloudera-scm-server/cloudera-scm-server.log

In the log shown at the end, there is a port that shows the port on which to start snooping.

Started ServerConnector@da518cb {SSL, [ssl, http/1.1]} {0.0.0.0 ssl 7183} Started ServerConnector@a77165b {HTTP/1.1, [http/1.1]} {0.0.0.0 ssl 7180} IV. Initialize Cloudera Manager

The browser opens http://172.18.8.200:7183, and the user name and password default to admin.

Receive permission.

Here we choose the free version, the toll version, please choose by yourself.

5. Cluster installation

Specify the node to add.

Select the repository, which we have already configured on the CM Server node.

Do not check, choose our own installation of JDK.

Provide SSH login credentials.

Install agents, because we have already installed it before, so it will be very fast here.

Install the selected Parcel, which we downloaded and configured / opt/cloudera/parcel-repo on the CM Server node.

Check the correctness of the host.

VI. Cluster setup

Choose the service you want to install and install the software according to your own needs.

Customize role assignments.

Database settings.

Review the changes, if there are specific directory settings or parameter settings, you can correct them here.

Run for the first time.

The installation is complete.

7. Manage clusters

Home page Dashboard.

HDFS .

HBase .

Thank you for your reading, the above is the content of "the installation process of Cloudera Manager and CDH 6.0.1 under CentOS7". After the study of this article, I believe you have a deeper understanding of the installation process of Cloudera Manager and CDH 6.0.1 under CentOS7. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.