Shulou(Shulou.com)06/01 Report--

The analysis and solution of OpenStack paralysis, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

If it is software, there will be bug,OpenStack. As long as you use it, you will encounter problems. Troubleshooting (troubleshooting) is an important skill of operation and maintenance OpenStack and other open source projects. When you encounter problems, you must use the strength of the community to locate, search, analyze and solve problems.

Below CloudMan will share a real case to restore the process of Troubleshooting at that time, hoping to give you some inspiration.

Problem description

One day, the customer's OpenStack is suddenly paralyzed: no operation can be completed normally, it has been in a state of execution, and there is no error on the interface, but the operation cannot be completed.

Analysis of problems

This is an overall problem. First, check the nova log and no errors are reported. Then look at the MySQL and RabbitMQ logs and find a large number of repeated errors in RabbitMQ:

I can't find this exchange in reply_529af7a7c3784c2d9dc5e72c603024a5 all the time. These reply_XXX are maintained by OpenStack itself, before running well, why suddenly can not find, should be an exception, has nothing to do with the configuration, it is estimated to be bug.

Let's google first. Search technical problems, google is the first choice, can not climb the wall to use bing, du Niang, or let her focus on Chinese: -)

Bing's search results are posted here:

The second one seems to be more reliable. Click in and find that the situation is exactly the same as ours, and a related bug is also mentioned.

Take a look at the content of bug, it is indeed a problem we have encountered, this is an oslo.messaging bug, and it is already fix.

Because the customer's OpenStack version is kilo, click the review link corresponding to kilo to see what changes have been made to fix.

A total of two files have been changed, click on the link of amqpdriver.py, you can see diff.

Comparing the contents of the customer's system / usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py file, it is indeed the previous version of fix.

The problem is identified, and there is a solution: update the olso.messageing package.

Solve the problem

The source code for OpenStack is maintained on github, and each module has its own repository. Oslo.messageing 's project home page is https://github.com/openstack/oslo.messaging

Since our current version is kilo, we are looking for the latest version of oslo.messaging on kilo.

In Tags, we see that kilo-eol,eol means "end of life", which is the final version of kilo.

Again, kilo-eol does contain the fix we want. The rest of the work is straightforward:

Download the oslo.messaging code base.

Install the kilo-eol version.

Restart related OpenStack related services.

Because oslo.messaging is the basic component and almost all services are used, you have to update each node and restart OpenStack. Although the workload was heavy, the problem was finally solved.

This is the end of the analysis and solution of OpenStack paralysis. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.