Shulou(Shulou.com)06/03 Report--

Error report:

Master.HMaster: Failed to become active master

Environment:

OS: CENTOS6.5

Hadoop: 2.7.1

Hbase: 1.0.4

ZooKeeper: 3.4.6

Action:

Hbase upgrade version 1.0.4-> 1.2.4

In the afternoon, I received a phone call from an old friend saying that Hbase Master could not be opened, because it was a production environment, so I was very anxious. At more than 4: 00 in the morning, the hard drive of a server broke, causing the server's Hadoop service to be abnormal and Hbase was also affected. It should be that the process in which the program was writing hbase was affected, and then the Hbase cluster hung up. After the administrator received the call from the production line, × × connected to the server to restart the service. It is normal for hadoop to be enabled, but the Master of Hbase has been unable to restart. And report an error, as follows:

After referring to many articles, we have been unable to solve this problem. In general, there are two ways to solve the problem of Hbase. Either there is a problem with memory allocation or a parameter exception. This is obviously not easy. We have suspected whether it is a Zookeeper exception, resulting in the inability to obtain ClusterID. However, if the Zookeeper exception, hadoop cannot be opened, so we rule out the Zookeeper exception.

Finally, we can see that it is mentioned on google that it is the influence of bug. In the end, there are two options: either upgrade the Hbase version or clean up and reconstruct the data. In contrast, updating the Hbase version is fast and secure, because the data of Hbase is stored in Hadoop, and there is no problem with the data of Hadoop, so upgrading Hbase will not affect the data of Hadoop.

As it was nearly dinnertime, I went out to have a meal. Before I finished packing, I received good news that the database cluster was open, that the data had not been lost, and that everything was fine.

The whole resolution process took nearly 4 hours, spent a lot of time studying the problem, tried a lot of methods, and finally upgraded the database cluster version for the first time. Bug this kind of thing, will not often encounter, but encounter is the most depressed time, either bypass, or solve head-on. This time we solved the problem head-on. Yeah.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.