Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to restore the / var/lib directory of the cm server node in cdh5.14 after it was deleted by mistake. Many people may not know much about it. In order to make you understand better, the editor summarized the following content for you. I hope you can get something according to this article.

The author encountered a sudden problem. In an online cdh cluster (cdh6.14), the / var/lib directory of the node where the cm server is located was mistakenly deleted. The erroneous deletion operator has tried to copy the / var/lib directory from the cm agent node cdh3 to the / var/lib directory of the cm server node cdh2, and then restart the cm server, and the restart is indeed successful, but after the restart, there are a large number of alarms on the cm page, both at the host level and at the service level. For example, the cdh2 host is in poor health, the host monitor on cdh2 failed to start, and several attempts to restart the cm cluster did not solve the problem. At the same time, services such as hdfs,hive,hbase,zookeeper are unstable after restart. Cm sometimes shows that these services are normal, and sometimes there are problems, but check hdfs web ui and find that hdfs services are normal.

After receiving the reported problem, the author first carefully observed the alarm information and host page in cm, and found that the hosts page sometimes shows cdh2 nodes and sometimes does not; at the same time, the roles of the original cdh2 nodes, such as host monitor,hdfs namenode, are sometimes shown to be running on cdh3 nodes, so it is clear that there is confusion when cm detects hosts, confusing cdh2 nodes and cdh3 nodes!

Principle analysis: the configuration files of the cm5.14 version are in / etc/cloudera-scm-agent and / etc/cloudera-scm-server by default and have not been deleted by mistake At the same time, the underlying mysql of hive and the underlying mysql of cm (which may or may not be the same database instance) have not been mistakenly deleted, so after restoring the files in the / var/lib directory of cm server, you can restart cm server successfully. At the same time, when cm server restarts, you can still find the MySQL Metabase through the configuration file / etc/cloudera-scm-server/db.properties and load the information in the Metabase. So you can still start the cm cluster. Directory to store CM agent state that persists across instances of the agent process and system reboots. Particularly, the agent's UUID is stored here.) in the problem environment, because the configuration item specifies / var/lib/cloudera-scm-agent, the cdh2 node restores the directory from another node cdh3 after being deleted by mistake. After recovery, the value stored in the uuid file in the cdh2 directory is the same as the value stored in the uuid file in the directory in the cdh3 node. meanwhile

Cm stores some metadata information for all Hosts nodes of the cm cluster in the cm.hosts table of the MySQL Metabase, including the uuid stored in the HOST_IDENTIFIER field. The uuid of the chd1 node stored in the week of the cm.hosts table in the problem environment is inconsistent with the uuid information stored in the / var/lib/cloudera-scm-agent/uuid file on the cdh2 node (because the uuid file is copied from the cdh3 node), which causes confusion when cm detects the cdh2 node and the cdh3 node, which triggers various alarms. In fact, the / var/lib/cloudera-scm-agent/cm_guid file stores the id of cm, which is the same as the value of the guid field in the cm.versions table, and the content of each cm node is the same; / var/lib/cloudera-scm-agent/uuid stores the globally unique id of the host, which is the same as the value of the HOST_IDENTIFIER field in the cm.hosts table, and the content of each node is different. View the contents of the uuid file: more / var/lib/cloudera-scm-agent/uuid; view the node metadata information stored in the cm.hosts table in MySQL metadata statement: select HOST_ID,HOST_IDENTIFIER,NAME,IP_ADDRSS,RACK_ID,CLUSTER_ID from cm.hosts.

Solution to the problem: modify the / var/lib/cloudera-scm-agent/uuid file of the cdh2 node, change it to the value of the recorded HOST_IDENTIFIER field of the node cdh2 in the cm.hosts table, and then restart cm-server and agent. The problem is finally solved. After the solution, the cdh2 node is normal, and the scm service roles on the node, such as host monitor, and big data service roles on the node, such as namenode, resource manager, zk, hbase master, etc., are also normal.

Summary of the usage of related directories in cm:

Cm-related configuration file directory:

/ etc/cloudera-scm-agent

/ etc/cloudera-scm-server

Working directories related to cm:

/ var/lib/cloudera-scm-server

/ var/lib/cloudera-scm-agent

/ var/lib/cloudera-host-monitor

/ var/lib/cloudera-service-monitor

Cm-related log directories:

/ var/log/cloudera-scm-server

/ var/log/cloudera-scm-agent

/ var/log/cloudera-scm-firehose

After reading the above, do you have any further understanding of how to restore the / var/lib directory of the cm server node in cdh5.14 after being deleted by mistake? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.