Shulou(Shulou.com)06/03 Report--

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

1. Purpose of document writing

This document describes how to upgrade Cloudera Manager and CDH, through which you will learn the following:

1. How to upgrade Cloudera Manager during downtime

two。 How to upgrade CDH during downtime

3. How to perform CDH rolling upgrade without affecting cluster jobs

The document is mainly divided into the following steps:

Overview of 1.Cloudera upgrade

2.Minor version Cloudera Manager and CDH upgrad

3.Maintenance version rolling upgrade CDH

4. Task verification during rolling upgrad

5. Functional verification after the completion of cluster upgrade

This document focuses on CDH upgrades and is based on the following assumptions:

The 1.CDH5.4.3 environment has been set up and running normally

two。 Cluster services (CM, HBase, HDFS, Hive, Hue, Oozie, Spark, Yarn, Zookeeper)

3.Cloudera Manager is installed using rpm

4.CDH is installed using parcels

5. Kerberos is not configured in the cluster

The following is the test environment, but it is not the hard limit of this manual:

1. Operating system: Redhat6.5

two。 Old CM/CDH version: CM5.4.3/ CDH5.4.3

Target upgrade version of 3.CM/CDH: CM5.11.1/ CDH5.11.1

4. Ec2-user users with sudo authority to operate

Upgrade recommendations:

1. For Maintenance Version upgrades for CDH, you can use a rolling upgrade, such as 5.4.x to 5.4.y, which fixes bug without changing API and does not involve new features.

two。 For Minor Version upgrade of CDH, downtime upgrade or rolling upgrade (if the span is small) is recommended, such as 5.x.x to 5.y.y upgrade, this version upgrade is mainly to add new features and new API.

3. For the Major Version upgrade of CDH, it is recommended that the upgrade must be stopped, and the modification and upgrade of existing applications and jobs need to be fully tested and verified before the upgrade, and the downtime will be longer during the corresponding upgrade. This version upgrade is generally a major version upgrade of Hadoop-related components, which may change API.

Overview of 2.Cloudera upgrade

The process described below applies to clusters managed by Cloudera Manager. For cluster upgrades that are not managed by Cloudera Manager, please refer to: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh\_ig\_upgrade\_command\_line.html#xd\_583c10bfdbd326ba--5a52cca-1476e7473cd--7f99

You can use tarballs or operating system packages to upgrade Cloudera Manager, and then packages or parcels to upgrade CDH. You may also need to install a new version of JDK. When upgrading Cloudera Manager, Cloudera Navigator will also be upgraded.

CDH and Cloudera Manager do not need to be upgraded at the same time, but you need to ensure that the Cloudera Manager and CDH versions are compatible. Cloudera Manager can manage current or previous major versions of CDH, or the same minor version of CDH, or lower minor versions of CDH. For example, Cloudear Manager 5. 7. 1 can manage CDH5.7.2,CDH5.6.1 and CDH4.8.6, but not CDH5.8.1. Cloudera Manager5.x cannot manage CDH3.x.

There are several specific options for upgrading:

2.1 assess the impact of the upgrade

It is generally necessary to plan a maintenance window long enough (downtime) to upgrade. Depending on the components you need to upgrade, the number of nodes in the cluster, and different hardware conditions, you may need to upgrade all day. Before you start the upgrade, you need to prepare some prerequisites and backup key data, which will also be described in this document when describing the upgrade steps.

Before upgrading, you need to consult Cloudera Manager's https://www.cloudera.com/documentation/enterprise/release-notes/topics/rg\_release\_notes.html for API changes, deprecated features, new features, and incompatible changes. You also need to check https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn\_consolidated\_pcm.html to confirm supported operating systems, JDK, databases, and other components.

There are three versions of the upgrade: major,minor and maintenance.

Major upgrade

Upgrades to the Major version typically have the following characteristics:

Large version of Hadoop changes, involving a lot of updates

Incompatible data formats major changes in the Cloudera Manager interface Cloudera Manager's database schema changes, but can be automatically handled during the upgrade process requiring longer downtime to redeploy the client

You can also upgrade only the major version and leave the minor version unchanged, for example, you can upgrade from 4.8.1 to 5.8.0. To facilitate the next version of major upgrade (CDH6), we recommend that you upgrade your cluster to 5.x now

Minor version upgrade

Minor version upgrade refers to upgrading the minor version based on the same major version, such as upgrading from 5.4.x to 5.8.x, which generally has the following characteristics:

New feature Bug repair possible database schema changes will be automatically processed when Cloudera Manager is upgraded

In general, upgrades to the minor version do not include incompatible changes or changes in data formats. Client configuration (https://www.cloudera.com/documentation/enterprise/latest/topics/cm\_mc\_client\_config.html#cmug\_topic\_5\_9) will be reset.

Maintenance version upgrade

Maintenance version upgrades are mainly major bug fixes or solutions to some security issues. There will be no compatibility modifications and new features.

Overview of 2.2Cloudera Manager upgrade

Upgrading Cloudera Manager includes the following steps:

1. Upgrade Cloudera Manager Server on the CM node, using operating system commands, such as Redhat's yum. You can also upgrade manually through tarballs. Tarballs upgrades are suitable for clusters that already have scripts, programs, or tools that can operate multiple machines at the same time.

two。 The upgrade wizard to upgrade Cloudera Manager agent,Cloudera Manager on all hosts can help you upgrade agent (JDK upgrade is optional), as well as manually upgrade JDK and agent through tarballs. CDH will not be upgraded during this process.

For Cloudera Manager upgrade, please refer to the official website document: https://www.cloudera.com/documentation/enterprise/latest/topics/cm\_upgrade.html#concept\_q1q\_mbs\_wx

Overview of 2.3CDH upgrade

CDH upgrades include upgrades to Hadoop-related components. You can use Cloudera Manager to upgrade CDH, either parcel or package.

2.3.1 upgrade using Parcel (also applicable to rolling upgrades)

We recommend using parcels to upgrade CDH because Cloudera Manager manages the automatic download, distribution and activation of these parcels. There are two ways to upgrade:

Parcels: you need to restart the cluster to complete the upgrade. Rolling upgrade: if HDFS has highly available HA enabled, you can perform a rolling upgrade without restarting the cluster. To simplify the upgrade steps, consider switching from package to parcels, so that Cloudera Manager can greatly simplify the upgrade steps. You can also switch from package to parcels when upgrading CDH5.

2.3.2 upgrade using Packages

Using packages to upgrade requires you to download the packages that needs to be upgraded in advance, and then manually run the package update command to upgrade, note that all hosts need to do the same.

Starting with Cloudera Manager5.3, CDH provides guided upgrades including major version upgrades (CDH5 to CDH5), minor version upgrades (CDH5.x to 5.y), and maintenance version upgrades (CDHa.b.x to CDHa.b.y). It is supported whether you use parcels installation or package, but package requires you to install and upgrade manually, but parcels can be automatically installed and upgraded by Cloudera Manager.

Overview of 2.4Cloudera Navigator upgrade

When upgrading Cloudera Manager, Cloudera Navigator Metadata and Audit servers are automatically upgraded. You can also upgrade other Navigator components such as Cloudera Navigator Key Trustee Server, Cloudera Navigator Key HSM and Cloudera Navigator Encrypt. Of course, you can choose not to upgrade.

2.5JDK upgrade

Before upgrading Cloudera Manager and CDH, make sure that all hosts in the cluster are using the supported Oracle JDK. See https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn\_consolidated\_pcm.html. Meanwhile, all hosts must use the same version of JDK: reference: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh\_cm\_upgrading\_to\_jdk8.html#xd\_583c10bfdbd326ba-590cb1d1-149e9ca9886--7c46

3.Minor version upgrade

Upgrade prerequisites:

1. Make sure that the current system version supports the new versions CDH5 and Cloudera Manager5 that you are upgrading to

two。 The original version of Cloudera Manager and CDH is 5.4.3

The target upgrade version for 3.Cloudera Manager and CDH is 5.11.1

3.1Cloudera Manager upgrade

This section focuses on how to upgrade Cloudera Manager using the RPM package.

3.1.1 preparation before upgrad

Yum source configuration for 3.1.1.1Cloudera Manager

1. Prepare to upgrade the rpm package

Wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/cloudera-manager-agent-5.11.1-1.cm5111.p0.9.el6.x86_64.rpmwget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el6.x86_ 64.rpmwget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/cloudera-manager-server-5.11.1-1.cm5111.p0.9.el6.x86_64.rpmwget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/cloudera-manager-server-db-2-5.11.1-1.cm5111.p0. 9.el6.x86_64.rpmwget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/enterprise-debuginfo-5.11.1-1.cm5111.p0.9.el6.x86_64.rpmwget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpmwget http://archive.cloudera.com / cm5/redhat/6/x86_64/cm/5.11.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

two。 Put the downloaded installation package in the / var/www/html/cm5.11.1 directory

Enter the cm5.11.1 directory and run the command:

[ec2-user@ip-172-31-8-141 cm5.11.1] $sudo createrepo.

3. Verify that it can be accessed through the browser

4. Configure / etc/yum.repos.d/cloudera-manager.repo file, which is as follows

[cloudera-manager] name = Cloudera Manager, Version 5.11.1baseurl = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.11.1/gpgcheck = 0

5. Test whether the Yum source is normal

[ec2-user@ip-172-31-8-141 yum.repos.d] $sudo yum clean all [ec2-user@ip-172-31-8-141 yum.repos.d] $sudo yum repolist

3.1.1.2 backing up Cloudera Manager database

1. Stop the Cloudera Management Service service

two。 Back up the Cloudera Manager database and execute the following command on the command line

L Cloudera Manager Server

L Cloudera Navigator Audit Server

L Cloudera Navigator Metadata Server

L Activity Monitor

L Reports Manager

[ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u cm-p-- databases cm > cm.dump [ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u nas-p-databases nas > nas.dump [ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u nms-p-databases nms > nms.dump [ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u am- P-- databases am > am.dump [ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u rm-p-- databases rm > rm.dump

To find information about the database, you can find it from the following files

Database information of ClouderaManager Server, which is obtained through configuration file / etc/cloudera-scm-server/db.properties

Database information of other services can be obtained through the CM management interface

3.1.2 upgrade steps

1. Stop the Cloudera Management Service service

two。 Stop Cloudera Manager Server and agent

Stop the cloudera-scm-agent service for all servers

[ec2-user@ip-172-31-8-141141] $sudo service cloudera-scm-agent stop

On the host running the Cloudera Manager server, stop the cloudera-scm-server service

[ec2-user@ip-172-31-8-141 cdh-shell-master] $sudo service cloudera-scm-server stop

3. Back up the following directories on Cloudera Manager

[ec2-user@ip-172-31-8-141 cm_bak] $sudo scp-r / etc/cloudera-scm-server/ / home/ec2-user/upgrade/cm_bak/ [EC2-user@ip-172-31-8-141 cm_bak] $sudo scp-r / etc/cloudera-scm-agent/ / home/ec2-user/upgrade/cm_bak/

Note that the file directory permissions are the same as the original data directory permissions.

4. Run the following command to upgrade Cloudera Manager Server

[ec2-user@ip-172-31-8-141 cm_bak] $sudo yum-y upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent

Check whether the installation is successful

[ec2-user@ip-172-31-8-141 cm_bak] $rpm-qa | grep cloudera-

5. Start the Cloudera Manager Server service

[ec2-user@ip-172-31-8-141 cloudera-scm-agent] $sudo service cloudera-scm-server start

6. Log in to the Cloudera Manager Management console to display the upgrade wizard

7. Choose to upgrade the Cloudera Manager Agent package to continue

8. Select Custom Repository, configure the address for Section 2.1.2, and click continue

9. Select JDK Select, click continue

10. Enter SSH login credentials as required, and click "continue"

11. Wait for the installation to complete, click continue

twelve。 Check complete, click "finish"

13. Enter the interface of the Audit changes Wizard and click continue

14. Select restart Cloudera Management Service and click "continue"

15. Service restart succeeded. Click "finish".

The cluster service is allowed to run normally

Cloudera Manager reports outdated configuration after upgrade, please restart the cluster service and redeploy the client configuration

3.1.3Cloudera Manager upgrade verification

1. View Cloudera Manager version

The version is displayed as the upgraded version

two。 Verify that Agent sends a heartbeat to Cloudera Manager

By default, the heartbeat is 15 seconds.

3. Check all hosts

4. Cluster historical monitoring data

3.2CDH upgrade

3.2.1 preparation before upgrad

3.2.1.1 prepare the parcels package for CDH

1. Select the cdh6.11.1 version to download the corresponding parcels

Wget http://archive.cloudera.com/cdh6/parcels/5.11.1/CDH-5.11.1-1.cdh6.11.1.p0.4-el6.parcelwget http://archive.cloudera.com/cdh6/parcels/5.11.1/CDH-5.11.1-1.cdh6.11.1.p0.4-el6.parcel.sha1wget http://archive.cloudera.com/cdh6/parcels/5.11.1/manifest.json

two。 Put the three downloaded files in the / var/www/html/cdh6.11.1 directory

3. Test whether it can be accessed properly through http

After the configuration is completed, it can be used to upgrade the CDH later.

3.2.2 upgrade proc

1. Stop the cluster service

two。 Backing up HDFS Metastore on NameNode

Back up the directory on the NameNode node to the specified directory

[ec2-user@ip-172-31-8-141 dfs] $cd / dfs/ [ec2-user@ip-172-31-8-141 dfs] $sudo tar-czvf / home/ec2-user/upgrade/nn_bak/nn_backup.tar.gz nn/

3. Backup database

L Cloudera Manager Server

[ec2-user@ip-172-31-8-141 hadoopdb_bak] $mysqldump-u hue-p-- database hue > hue.dump [ec2-user@ip-172-31-8-141 hadoopdb_bak] $mysqldump-u hive-p-database metastore > metastore.dump [ec2-user@ip-172-31-8-141 hadoopdb_bak] $mysqldump-u oozie-p-database oozie > oozie.dump [ec2-user@ip-172-31-8-141 hadoopdb_bak] $mysqldump-u sentry- P-- database sentry > sentry.dump [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u cm-p-databases cm > cm.dump [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u nas-p-databases nas > nas.dump [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u nms-p-databases nms > nms.dump [ec2-user@ip-172-31-8-141cmdb_] Bak] $mysqldump-u am-p-- databases am > am.dump [ec2-user@ip-172-31-8-141 cmdb_bak] $mysqldump-u rm-p-- databases rm > rm.dump

4. Add a new version of the CDH repository to the cluster

5. Run the upgrade Wizard

6. Enter the upgrade wizard interface

7. Confirm that the CDH version is correct and click "continue"

8. Select "Yes" and click "continue"

9. Click "continue"

10. After waiting for the installation to complete, click continue

11. When the check host is complete, click "continue"

twelve。 Select the full cluster upgrade and click "continue"

13. Wait for the upgrade cluster command to complete, click "continue"

14. Cluster upgrade succeeded

15. Final metadata upgrade

Before finalizing the metadata, run for a few days or even weeks to see if the cluster is normal, and then finalize the metadata after finding that there are no anomalies in all tasks. Once the metadata is finalized, it cannot be rolled back to the older version unless there is a backup of the data. Perform a final metadata upgrade operation on both the master and slave nodes of NameNode:

3.2.3 functional verification

1. Run a MapReduce job

Log in to the cluster server and perform the following instructions

Sudo-u hdfs hadoop jar / opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

two。 Testing with Hue

Log in to Hue normally, operate HBase through Hue, and create a new test table in HBase.

Add a piece of data to the test table

Data added successfully

Check whether the table exists through hbase-shell

3. Test hive

Beeline logs in to hive for testing

Hue Test hive

Can query test table data normally

Rolling upgrade of Maintenance version of 4.CDH

Upgrade prerequisites:

Make sure that the current system version supports the new version you are upgrading to, CDH5 and Cloudera Manager5. The original Cloudera Manager and CDH versions are 5.10.0Cloudera Manager and CDH. The target upgrade version is 5.10.2 Cluster enabled NameNode High availability Cluster enabled Resource Manager High availability

4.1Cloudera Manager upgrade

4.1.1 preparation before upgrad

Yum source configuration for 4.1.1.1Cloudera Manager

1. Download the RPM installation package for Cloudera Manager

Sudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/cloudera-manager-agent-5.10.2-1.cm5102.p0.9.el6.x86_64.rpmsudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/cloudera-manager-daemons-5.10.2-1.cm5102.p0.9.el6. X86_64.rpmsudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/cloudera-manager-server-5.10.2-1.cm5102.p0.9.el6.x86_64.rpmsudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/cloudera-manager-server-db-2-5.10.2-1. Cm5102.p0.9.el6.x86_64.rpmsudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/enterprise-debuginfo-5.10.2-1.cm5102.p0.9.el6.x86_64.rpmsudo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/jdk-6u31-linux-amd64.rpmsudo wget http: / / archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10.2/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

two。 Put the downloaded 7 RPM installation packages in the / var/www/html/cm5.10.2 directory

3. Execute the following command in this directory

[ec2-user@ip-172-31-8-141 cm5.10.2] $sudo createrepo.

4. Test whether it can be accessed properly through the browser.

5. Modify / etc/yum.repos.d/cloudera-manager.repo configuration

[ec2-user@ip-172-31-8-141 cm5.10.2] $sudo vim / etc/yum.repos.d/cloudera-manager.repo [cloudera-manager] name = Cloudera Manager, Version 5.10.2baseurl = http://ip-172-31-8-141.ap-southeast-1.compute.internal/cm5.10.2gpgcheck = 0

6. Test whether the yum source of Cloudera Manager is configured

[ec2-user@ip-172-31-8-141 cm5.10.2] $sudo yum clean all [ec2-user@ip-172-31-8-141 cm5.10.2] $sudo yum repolist

4.1.1.2 backing up the Cloudera Manager database

1. Stop Cloudera Management Service service and Cloudera ManagerService service

Stop the ClouderaManagement Service service through the Cloudera Manager console

Stop the Cloudera Manager Server service with the following command

[ec2-user@ip-172-31-8-141141] $sudo service cloudera-scm-server stop

two。 Back up the Cloudera Manager database and execute the following command on the command line

Cloudera Manager ServerActivity MonitorReports Manager [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u cm-p-- databases cm > cm.dump [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u am-p-databases am > am.dump [ec2-user@ip-172-31-8-141cmdb_bak] $mysqldump-u rm-p-- databases rm > rm.dump

4.1.2 upgrade proc

1. Make sure the Cloudera Management Service service is stopped

two。 Stop the Agent service by executing the following command on all hosts where Cloudera Manager Agent is installed

[ec2-user@ip-172-31-8-141 cdh-shell-master] $sudo service cloudera-scm-agent stop

3. Submit a MR job to the cluster

4. Run the following command to upgrade Cloudera Manager Server

[ec2-user@ip-172-31-8-141 cm_bak] $sudo yum-y upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent

Check whether the installation is successful

[ec2-user@ip-172-31-8-141 cm_bak] $rpm-qa | grep cloudera-

5. Start the Cloudera Manager Server service with the following command

[ec2-user@ip-172-31-8-141 cdh-shell-master] $sudo service cloudera-scm-server start

6. Log in to the Cloudera Manager console and enter the display upgrade wizard

7. Select upgrade Cloudera Manager Agent and click "continue"

8. Enter the Cloudera Manager repository URL and click continue

9. Check the java option and click "continue"

10. Enter SSH login credentials and click "continue"

11. Wait for the Cloudera Manager Agent upgrade to complete, click "continue"

twelve。 When the check is complete, click "continue"

13. Complete the upgrade and go to the upgrade wizard page

14. Make changes as needed, click "continue"

15. Restart Cloudera Management Service and click "continue"

16. Click "finish" to enter the Cloudera Manager console home page

The upgrade is successful, and the related services of Hadoop are normal.

17. Upgrade completed MR job is free to run

18. The job ran successfully

19. Statistical results

TypeDateTimeInput_data_sizeDuration (s) Throughput (bytes/s) Throughput/nodeHadoopWordcount2017/8/1712:02:3932849063607991.5023313060733130607

4.1.3 functional verification

1. View Cloudera Manager version

The version displayed as 5.10.2 indicates that the upgrade was successful.

two。 Verify that Agent sends a heartbeat to Cloudera Manager

By default, the heartbeat is 15 seconds.

3. Check all hosts

4. Cluster historical monitoring data

The historical monitoring data is normal and there is no data only during the upgrade Cloudera Manager Agent period.

4.2 Rolling upgrade CDH

4.2.1 preparation before upgrad

4.2.1.1 prepare the CDHparcels package

1. Select the cdh6.10.0 version to download the corresponding parcels

Wget http://archive.cloudera.com/cdh6/parcels/5.10.2/CDH-5.10.2-1.cdh6.10.2.p0.5-el6.parcelwget http://archive.cloudera.com/cdh6/parcels/5.10.2/CDH-5.10.2-1.cdh6.10.2.p0.5-el6.parcel.sha1wget http://archive.cloudera.com/cdh6/parcels/5.10.2/manifest.json

two。 Put the three downloaded files in the / var/www/html/cdh6.10.2 directory

3. Test whether it can be accessed properly through http

After the configuration is completed, it can be used to upgrade the CDH later.

4.2.1.2 run a MR job

1. Submit a MapReduce job for WordCount before upgrading the environment

two。 View the running status of the job

3. Job run completed

4. Statistical results

TypeDateTimeInput_data_sizeDuration (s) Throughput (bytes/s) Throughput/nodeHadoopWordcount2017/8/1711:09:2032849063607984.0963337993833379938

4.2.1.3 configure the repository URL for CDH

Enter the CDH repository URL configuration interface according to the identity

Configure CDH's repository URL through the Cloudera Manager console

To save your changes, click "check for new parcels"

Configuration was successful.

4.2.1.4 backing up HDFS metadata

Back up HDFS metadata information using the following command

[ec2-user@ip-172-31-8-141 upgrade] $sudo-u hdfs hadoop dfsadmin-fetchImage / home/ec2-user/upgrade/hdfs_metastore_bak

4.2.2 upgrade proc

1. Enter the Cluster upgrade Wizard through the Cloudera Manager console

two。 Select the CDH version and click "continue"

3. When checked, click "continue"

4. Cluster software check is normal, click "continue"

5. Wait for parcels to install successfully, click "continue"

6. Host check passed, click "continue"

7. Select rolling upgrade, set parameters as needed, and click "continue"

8. Submit MR jobs to the cluster during a rolling upgrade

The job is running normally

9. Wait for the rolling upgrade command to complete, click "continue"

10.CDH upgrade completed

11.MR job run completed

twelve。 Statistical results

TypeDateTimeInput_data_sizeDuration (s) Throughput (bytes/s) Throughput/nodeHadoopWordcount2017/8/1712:42:04328490636071042.4613151107131511071

4.2.3 functional verification

1. Go to the Cloudera Manager console home page to see if the cluster is normal.

two。 Submit a MR job to the cluster

Log in to the cluster server and perform the following instructions

Sudo-u hdfs hadoop jar / opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

The job is running normally

3. Testing with Hue

Log in to Hue normally and operate HBase through Hue to add a piece of data to the test table

Data added successfully

View new data through hbase-shell

4. Test Hive

Beeline logs in to hive for testing

Insert a piece of data into the test table

View test table data

Hue Test hive

Can query test table data normally, and the query result is consistent with the command line.

5. Use Hue to test Impala

Query test table

Command line query

Comparison of task execution time in the process of 4.3CDH upgrade

Execution phase TypeDateTimeInput_data_sizeDuration (s) Throughput (bytes/s) Throughput/node upgrade before HadoopWordcount2017/8/1711:09:2032849063607984.0963337993833379938 upgrade HadoopWordcount2017/8/1712:02:3932849063607991.5023313060733130607 in CM upgrade HadoopWordcount2017/8/1712:42:04328490636071042.4613151107131511071 in CDH

The use of job resources will not be affected in the process of upgrading CM.

During the rolling upgrade of CDH, the job time is increased due to the loss of computing resources caused by restarting the service.

Drunken whips are famous horses, and teenagers are so pompous! Lingnan Huan Xisha, under the vomiting liquor store! The best friend refuses to let go, the flower of data play!

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.