Shulou(Shulou.com)05/31 Report--

This article mainly introduces what to do if the dashboard that logs in to openstack can not start the CVM. It has certain reference value. Interested friends can refer to it. I hope you will learn a lot after reading this article. Let the editor take you to know about it.

Background:

The test machine shuts down every night, and there are always some inexplicable problems when it starts every morning. Log in to the dashboard of openstack and find that the CVM cannot be started and failed to create the cloud disk.

The solution is as follows: 1. Restore ceph cluster

Looking at the status of the underlying storage ceph, we found that several osd were not started, so the osd,ceph cluster that was started and shut down went into a healthy state.

The ceph cluster consists of three machines: 216,217,218,216,217,218,216,217,217,218,216,217,217,218,216,217,217,218,216,217,217,217,218,217,217,217,218,217,217,218,217,217,218,217,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,217,217,218,

2. If you failed to create a cloud disk, check the cinder log # view / var/log/cinder/volume.log

You see the following error:

. . .

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01lo 44.865 9622 ERROR cinder.utils [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Volume driver RBDDriver not initialized

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01 ERROR oslo_messaging.rpc.dispatcher 44.866 9622 [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Exception during message handling: Volume driver not ready.

2016-08-02 15 01lo 44.866 9622 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):

2016-08-02 15 line 01VR 44.866 9622 TRACE oslo_messaging.rpc.dispatcher File "/ usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _ dispatch_and_reply

2016-08-02 15 TRACE oslo_messaging.rpc.dispatcher executor_callback 01purl 44.866 9622))

2016-08-02 15 line 01type 44.866 9622 TRACE oslo_messaging.rpc.dispatcher File "/ usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", in _ dispatch

2016-08-02 15 TRACE oslo_messaging.rpc.dispatcher executor_callback 01purl 44.866 9622)

2016-08-02 15 line 01type 44.866 9622 TRACE oslo_messaging.rpc.dispatcher File "/ usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _ do_dispatch

2016-08-02 15 ctxt 01lo 44.866 9622 TRACE oslo_messaging.rpc.dispatcher result = func (ctxt, * * new_args)

2016-08-02 15 line 01 TRACE oslo_messaging.rpc.dispatcher File 44.866 9622 "/ usr/lib/python2.7/site-packages/osprofiler/profiler.py"

2016-08-02 15 kwargs 01lo 44.866 9622 TRACE oslo_messaging.rpc.dispatcher return f (* args, * *)

2016-08-02 15 line 01lo 44.866 9622 TRACE oslo_messaging.rpc.dispatcher File "/ usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1076, in initialize_connection

2016-08-02 15 01R 44.866 9622 TRACE oslo_messaging.rpc.dispatcher utils.require_driver_initialized (self.driver)

2016-08-02 15 line 01 TRACE oslo_messaging.rpc.dispatcher File 44.866 9622 "/ usr/lib/python2.7/site-packages/cinder/utils.py", in require_driver_initialized

2016-08-02 15 01purl 44.866 9622 TRACE oslo_messaging.rpc.dispatcher raise exception.DriverNotInitialized ()

2016-08-02 15 TRACE oslo_messaging.rpc.dispatcher DriverNotInitialized 01purl 44.866 9622: Volume driver not ready.

2016-08-02 15 purl 01 purl 44.866 9622 TRACE oslo_messaging.rpc.dispatcher

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01 INFO oslo_messaging._drivers.impl_rabbit 44.870 9622 [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Connecting to AMQP server on 192.168.217

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01VR 44.883 9622 INFO oslo_messaging._drivers.impl_rabbit [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Connected to AMQP server on 192.168.217

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01 ERROR oslo_messaging._drivers.common 44.885 9622 [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Returning exception Volume driver not ready To caller

2016-08-02 15 04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc 01VR 44.885 9622 ERROR oslo_messaging._drivers.common [req-0706d398-b2bf-4925-9613-04bdb1deb8f9 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] ['Traceback (most recent call last):\ nforth,' File "/ usr/

Lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py ", line 142, in _ dispatch_and_reply\ nexecutor_callback))\ nThe, 'File" / usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py ", line 186, in _ dispatch\ nexecut

Or_callback), 'File "/ usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _ do_dispatch\ nresult = func (ctxt, * * new_args)\ nfarmer,' File "/ usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105

, in wrapper\ n return f (* args, * * kwargs)\ nfarmer, 'File "/ usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1076, in initialize_connection\ nutils.require_driver_initialized (self.driver)\ nfarmer,' File "/ usr/lib/python2.7/

Site-packages/cinder/utils.py ", line 598, in require_driver_initialized\ nraise exception.DriverNotInitialized ()\ nfarmer, 'DriverNotInitialized: Volume driver not ready.\ n']

2016-08-02 15 4abc-a6a4 02 INFO cinder.volume.manager 42.248 9622 [req-fe77cb0b-b8e0-4abc-a6a4-3127ec2dcbc3 -] Updating volume status

2016-08-02 15 4abc-a6a4 02 WARNING cinder.volume.manager 42.250 9622 [req-fe77cb0b-b8e0-4abc-a6a4-3127ec2dcbc3 -] Unable to update stats, RBDDriver-1.1.0 driver is uninitialized.

2016-08-02 15 4ce4-a9aa-0d782076aca4 03 INFO cinder.volume.manager 42.249 9622 [req-eca34b20-dca4-4ce4-a9aa-0d782076aca4 -] Updating volume status

2016-08-02 15 4ce4-a9aa-0d782076aca4 03V 42.251 9622 WARNING cinder.volume.manager [req-eca34b20-dca4-4ce4-a9aa-0d782076aca4 -] Unable to update stats, RBDDriver-1.1.0 driver is uninitialized.

. . .

The most frequent occurrence is RBDDriver-1.1.0 driver is uninitialized.

Turning up the log can be traced back to when the machine was just started, and it was found that the RBDDriver-1.1.0 driver was not initialized because it could not connect to the ceph cluster.

2016-08-02 08 80fc-f18a6535ce90 41.014 9622 INFO cinder.volume.manager [req-72b8c8a4-a831-4395-80fc-f18a6535ce90 -] Starting volume driver RBDDriver (1.1.0)

2016-08-02 08 80fc-f18a6535ce90 52 ERROR cinder.volume.drivers.rbd 41.074 9622 [req-72b8c8a4-a831-4395-80fc-f18a6535ce90 -] Error connecting to ceph cluster.

2016-08-02 08 TRACE cinder.volume.drivers.rbd Traceback 52 TRACE cinder.volume.drivers.rbd Traceback 41.074 9622 (most recent call last):

2016-08-02 08 TRACE cinder.volume.drivers.rbd File 52V 41.074 9622 TRACE cinder.volume.drivers.rbd File "/ usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 314, in _ connect_to_rados

2016-08-02 08 TRACE cinder.volume.drivers.rbd client.connect 52V 41.074 9622

2016-08-02 08 TRACE cinder.volume.drivers.rbd File 52V 41.074 9622 TRACE cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969)

2016-08-02 08 TRACE cinder.volume.drivers.rbd TimedOut 5214 9622 TRACE cinder.volume.drivers.rbd TimedOut: error connecting to the cluster

2016-08-02 08 TRACE cinder.volume.drivers.rbd 52V 41.074 9622

2016-08-02 08 80fc-f18a6535ce90 52V 41.080 9622 ERROR cinder.volume.manager [req-72b8c8a4-a831-4395-80fc-f18a6535ce90 -] Error encountered during initialization of driver: RBDDriver

2016-08-02 08 80fc-f18a6535ce90 52 ERROR cinder.volume.manager 41.081 9622 [req-72b8c8a4-a831-4395-80fc-f18a6535ce90 -] Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.

2016-08-02 08 TRACE cinder.volume.manager Traceback 52 TRACE cinder.volume.manager Traceback 41.081 9622 (most recent call last):

2016-08-02 08 TRACE cinder.volume.manager File 41.081 9622 TRACE cinder.volume.manager File "/ usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 302, in init_host

2016-08-02 08 TRACE cinder.volume.manager self.driver.check_for_setup_error 52purl 41.081 9622 ()

2016-08-02 08 TRACE cinder.volume.manager File 52V 41.081 9622 TRACE cinder.volume.manager File "/ usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper

2016-08-02 08 TRACE cinder.volume.manager return 52V 41.081 9622 TRACE cinder.volume.manager return f (* args, * * kwargs)

2016-08-02 08 TRACE cinder.volume.manager File 52V 41.081 9622 "/ usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 287, in check_for_setup_error

2016-08-02 08 TRACE cinder.volume.manager with RADOSClient 52 TRACE cinder.volume.manager with RADOSClient 41.081 9622 (self):

2016-08-02 08 TRACE cinder.volume.manager File 52V 41.081 9622 TRACE cinder.volume.manager File "/ usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 242, in _ _ init__

2016-08-02 08 TRACE cinder.volume.manager self.cluster 52V 41.081 9622 TRACE cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados (pool)

2016-08-02 08 TRACE cinder.volume.manager File 52V 41.081 9622 TRACE cinder.volume.manager File "/ usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 322, in _ connect_to_rados

2016-08-02 08 TRACE cinder.volume.manager raise exception.VolumeBackendAPIException 52lo 41.081 9622 TRACE cinder.volume.manager raise exception.VolumeBackendAPIException (data=msg)

2016-08-02 08 TRACE cinder.volume.manager VolumeBackendAPIException 52 TRACE cinder.volume.manager VolumeBackendAPIException 41.081 9622: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.

2016-08-02 08 52purl 41.081 9622 TRACE cinder.volume.manager

2016-08-02 08 INFO oslo_messaging._drivers.impl_rabbit 52V 41.127 9622 INFO oslo_messaging._drivers.impl_rabbit [req-d845763e-483f-4495-8dc4-70fb2757d522 -] Connecting to AMQP server on 192.168.217 8dc4 5672

2016-08-02 08 INFO oslo_messaging._drivers.impl_rabbit 52V 41.146 9622 INFO oslo_messaging._drivers.impl_rabbit [req-d845763e-483f-4495-8dc4-70fb2757d522 -] Connected to AMQP server on 192.168.217 8dc4 5672

2016-08-02 08 INFO cinder.volume.manager 52V 42.163 9622 INFO cinder.volume.manager [req-8e733ab2-45c7-41c6-b89f-46ad5319d291 -] Updating volume status

2016-08-02 08 41c6-b89f-46ad5319d291 52 WARNING cinder.volume.manager 42.164 9622 [req-8e733ab2-45c7-41c6-b89f-46ad5319d291 -] Unable to update stats, RBDDriver-1.1.0 driver is uninitialized.

2016-08-02 08 44a7-a80f-c7da27d5074a 53 INFO cinder.volume.manager 42.163 9622 [req-89715cce-2cc6-44a7-a80f-c7da27d5074a -] Updating volume status

2016-08-02 08 44a7-a80f-c7da27d5074a 53 WARNING cinder.volume.manager 42.165 9622 [req-89715cce-2cc6-44a7-a80f-c7da27d5074a -] Unable to update stats, RBDDriver-1.1.0 driver is uninitialized.

2016-08-02 08 4800-bf86 54 INFO cinder.volume.manager 42.164 9622 [req-42e4bb17-c2b7-4800-bf86-9f80947d3ca9 -] Updating volume status

2016-08-02 08 WARNING cinder.volume.manager 54 9f80947d3ca9 42.165 9622 WARNING cinder.volume.manager [req-42e4bb17-c2b7-4800-bf86-9f80947d3ca9 -] Unable to update stats, RBDDriver-1.1.0 driver is uninitialized.

3. Restart the service of cinder

Although the cluster has returned to normal now, RBDDriver is still not working properly, so you need to restart the cinder service here.

# systemctl restart openstack-cinder-api openstack-cinder-scheduler openstack-cinder-volume

After restarting the service, wait a while and then check the log

2016-08-02 15 Caught SIGTERM 1018 08.805 1953 INFO cinder.openstack.common.service [req-01aa405c-6b1b-4456-b38b-22fc86a74ba9 -] stopping children

2016-08-02 15 Waiting on 1014 08.808 1953 INFO cinder.openstack.common.service [req-01aa405c-6b1b-4456-b38b-22fc86a74ba9 -] Waiting on 1 children to exit

2016-08-02 15 killed by signal 10 Child 08.809 1953 INFO cinder.openstack.common.service [req-01aa405c-6b1b-4456-b38b-22fc86a74ba9 -] 15

2016-08-02 15 861c-43ab-9b5a-d4b71453daeb 10 INFO cinder.volume.manager 10.046 59977 [req-2cae44c1-861c-43ab-9b5a-d4b71453daeb -] Determined volume DB was not empty at startup.

2016-08-02 15 861c-43ab-9b5a-d4b71453daeb 10 INFO cinder.openstack.common.service 10.066 59977 [req-2cae44c1-861c-43ab-9b5a-d4b71453daeb -] Starting 1 workers

2016-08-02 15 861c-43ab-9b5a-d4b71453daeb 10 INFO cinder.openstack.common.service 10.070 59977 [req-2cae44c1-861c-43ab-9b5a-d4b71453daeb -] Started child 60005

2016-08-02 15 version 10 14 60005 INFO cinder.service [-] Starting cinder-volume node (version 2015.1.2)

2016-08-02 15 4022-a52c-3a8ce3f1f5ae 10 INFO cinder.volume.manager 10.073 60005 [req-dbf976a6-1253-4022-a52c-3a8ce3f1f5ae -] Starting volume driver RBDDriver

2016-08-02 15 4022-a52c-3a8ce3f1f5ae 10.178 60005 INFO cinder.volume.manager [req-dbf976a6-1253-4022-a52c-3a8ce3f1f5ae -] Updating volume status

2016-08-02 15 INFO oslo_messaging._drivers.impl_rabbit 10.226 60005 INFO oslo_messaging._drivers.impl_rabbit [req-dbf976a6-1253-4022-a52c-3a8ce3f1f5ae -] Connecting to AMQP server on 192.168.8.217

2016-08-02 15 4022-a52c-3a8ce3f1f5ae 10.240 60005 INFO oslo_messaging._drivers.impl_rabbit [req-dbf976a6-1253-4022-a52c-3a8ce3f1f5ae -] Connected to AMQP server on 192.168.8.217

2016-08-02 15 424f-ba60 10 INFO oslo_messaging._drivers.impl_rabbit 10.290 60005 [req-c1532ac0-ef22-424f-ba60-02b1de8967c9 -] Connecting to AMQP server on 192.168.217

2016-08-02 15 424f-ba60 10 424f-ba60 10.303 60005 INFO oslo_messaging._drivers.impl_rabbit [req-c1532ac0-ef22-424f-ba60-02b1de8967c9 -] Connected to AMQP server on 192.168.217

2016-08-02 15 38dd-4812 11V 43.288 60005 INFO cinder.volume.flows.manager.create_volume [req-e2cf8138-38dd-4812-98d7-a8ec27b53800 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Volume 37b011a3-f5f2-4bb2-8cb6-e646bf955ce6:

Being created as image with specification: {'status': utilising creatingstones,' image_location': (None, None), 'volume_size': 1,' volume_name': u'volume-37b011a3-f5f2-4bb2-8cb6 Murray e646bf955ce6colors, 'image_id': u'5008d784-fa11-45d3-b910-653c7f2420fccolors,' I

Mage_service':, 'image_meta': {' status': upright activety, 'name': upright circs installed rawls,' deleted': False, 'container_format': upright activities,' created_at': datetime.datetime (2016, 7, 28, 6, 15, 2

8, tzinfo=), 'disk_format': upright qcow2,' updated_at': datetime.datetime (2016, 7, 28, 6, 15, 30, tzinfo=), 'id': u'5008d784-fa11-45d3-b910-653c7f2420fc61c20829e34bbeb67e9b8fc4b5a6b4,' min_ram': 0

'checksum': uprie1eca47dc88f4879d8a229cc70a07c6Austria, 'min_disk': 0,' is_public': True, 'deleted_at': None,' properties': {}, 'size': 13287936}}

2016-08-02 15 38dd-4812 11V 44.840 60005 INFO cinder.image.image_utils [req-e2cf8138-38dd-4812-98d7-a8ec27b53800 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Image download 12.00 MB at 12.00 MB/s

2016-08-02 15 38dd-4812 11V 45.864 60005 INFO cinder.image.image_utils [req-e2cf8138-38dd-4812-98d7-a8ec27b53800 6d8248aff00546bb86f439342780e770 a3f5a48418e449578f9c0c56105287cc -] Converted 12.00 MB image at 12.00 MB/s

4. It is normal to log in to the dashboard page, create a cloud disk and start the CVM. Thank you for reading this article carefully. I hope the article "what to do if you log in to the dashboard of openstack and find that you can't start the CVM" will be helpful to you. At the same time, I also hope you will support us and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.