Shulou(Shulou.com)06/01 Report--

This article shows you what is the solution to the CDH cluster upgrade accident, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

01 problem phenomenon

One day, brother Ming received help, and the cluster operation and maintenance staff encountered a problem when upgrading the cdh cluster through the upgrade cluster option of cloudera manager (see figure 1 below), and got stuck in the "Execute command Upgrade Hive Metastore Database Schema on service Hive" step. Because our application system is an offline computing operation based on Hive, the hive upgrade is not successful, and the application can not be started, this problem needs to be solved urgently.

The steps before the "Execute command Upgrade Hive Metastore Database Schema on service Hive" step were performed successfully, including the upgrade of HDFS:

02 problem analysis

View the stdout and stderr logs when "Execute command Upgrade Hive Metastore Database Schema on service Hive" is performed on the cm page. The contents are as follows (the page log didn't have time to take screenshots. The following screenshots are the contents of the page logs copied at that time):

Stdout.log:

Stderr.log:

View the contents of the hive metastore database upgrade script:

It can be seen that the upgrade script simply changes the mysql metastore db of hive, updates the structure of the DBS table in the hive library (with a column create_time added), and updates a record of the cdh_version table.

The execution of such a simple ddl statement and dml statement will fail, most likely because the database username and password are incorrect or do not have update permissions. We need to verify that the user name and password used are correct, and that we have the appropriate permissions. These connection information and username and password information are specified on the hive configuration page in cm:

Unfortunately, as you can see in the figure above, the cm page masks the user's configured hive metastore db password in order to protect sensitive information.

(in fact, the connection information and user name information are also told in the stdout.log log, which is consistent with the above configuration: Metastore connection URL is jdbc:mysql://node1:3306/hive?useUnicode=true&characterEncoding=UTF-8;Metastore connection User and root.)

So next, our idea is to get the correct user name and password that hive metastore db needs to use, and retry the cluster upgrade step again under the above page configuration.

03 how to get the user name and password of hive metastore db?

How do I get the user name and password of hive metastore db?

Method 1: ask the cluster administrator for it.

This method is feasible, but because the user name and password of hive metastore db are set when the cluster is built for the first time, the user name and password are rarely used in normal use after the cluster is built, and the longer the day goes on, the cluster administrator may forget the original user name and password. (there may be confusion if there are too many clusters.)

Method 2: check the configuration file hive-site.xml to get.

This method was feasible in earlier versions of hive, but after 1.x (I don't remember the specific version.) Also to protect sensitive information, this information is also blocked in hive-site.xml, as shown below:

Method 3: access the specific restful interface provided by cm through curl to obtain cluster configuration information, including hive metastore username and password.

The address and access method of this API is: curl-u cm-user:cm-pwd "http://cm-host:7180/api/v19/cm/deployment". Examples are as follows:

04 problem solving

After obtaining the correct user name and password of hive metastore db through method 3 above, and after configuring the correct user name and password on the hive configuration page, try to upgrade the cluster again (by resuming the upgrade cluster command on the Running Commands page of cm), and the cluster upgrade is executed correctly until the end, as shown below:

The end of 05HDFS cluster upgrade

Partners who have done hdfs cluster upgrade will notice that after upgrading the cluster through the above steps, the hdfs cluster can provide read and write services normally, but there will be a message "upgrade in progress. Not yet finalized" on the web page of hdfs, as shown below:

In fact, this is the upgrade mechanism intended by hdfs: hdfs allows users to fully test and verify the cluster again after the cluster has been upgraded to ensure that everything is correct before coming to finalize, that is, if the entire update; of commit fails to verify, it can be rolled back to the pre-upgrade version. "Finalizing upgrade is separate process that should be invoked manually after you confirm that data migration went well."

We execute the command hdfs dfsadmin-finalizeUpgrade to upgrade the finalize:

Check the hdfs web ui again and the message is missing:

06key take-aways

Summarize the knowledge points:

1. The user name and password of hive metastore db can be obtained from the following command:

Curl-u cm-user:cm-pwd "http://cm-host:7180/api/v19/cm/deployment"

2. After the hdfs cluster upgrade is completed, you can perform hdfs dfsadmin-finalizeUpgrade to confirm the upgrade, or roll back to the previous version.

The above content is what is the solution to the CDH cluster upgrade accident? have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.