Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to frequently add and delete osd that leads to osd not being able to up. Many people may not know much about it. In order to make you understand better, the editor summarized the following content for you. I hope you can get something according to this article.

# Environment introduction

Pre-launch system, crushmap has been manually set up, and the location where the osd.139 is located has been specified

Noout (ceph osd set noout) enabled in the cluster

Ceph version: 0.94.5

Osd sets osd crush update on start = false to avoid changing crushmap after osd starts.

# failure phenomenon in the process of simulating a single node failure, the same osd is manually added and deleted many times (only data and keyring are deleted, and crushmap content is not moved). Finally, it is found that although the newly added osd process has been started and no error has been reported in the startup log, it is always unable to enter the up status.

2016-04-01 11 Vera 199764da52395923e0b32908d83a9f7304401fee43 16.868837 7fee3654b900 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 104255.2016-04-01 11 9764da52395923e0b32908d83a9f7304401fee43 9764da52395923e0b32908d83a9f7304401fee43 1919.295992 7fee3654b900 0 osd.139 12789 crush map has features 2200130813952, adjusting msgr requires for clients2016-04-01 11V 19V 19.296008 7fee3654b900 0 osd.139 12789 crush map has features 2200130813952 was 8705, adjusting msgr requires for mons2016-04-01 11 Fringe 1919.296016 7fee3654b900 0 osd.139 12789 crush map has features 2200130813952 Adjusting msgr requires for osds2016-04-01 11 osd.139 1919 19296052 7fee3654b900 0 osd.139 12789 load_pgs2016-04-01 11 osd.139 1919296094 7fee3654b900 0 osd.139 12789 load_pgs opened 0 pgs2016-04-01 11 Vera 19.296878 7fee3654b900-1 osd.139 12789 log_to_monitors {default=true} 2016-04-01 11RV 19.305091 7fee246f1700 0 osd.139 12789 ignoring osdmap until we have initialized2016-04-01 11Ranger 1919.305239 7fee246f1700 0 osd.139 12789 ignoring osdmap until we have Initialized2016-04-01 11 osd.139 1919.305425 7fee3654b900 0 osd.139 12789 done with init Starting boot process

When you turn on debug osd=20, you find that you always do the following

2016-04-01 11 update_osd_stat osd_stat 46 update_osd_stat osd_stat 23.300790 7f9219d15700 20 osd.139 12813 update_osd_stat osd_stat (538 MB used, 3723 GB avail, 3724 GB total, peers [] / [] op hist []) 2016-04-01 11 Vera 46V 23.300821 7f9219d15700 5 osd.139 12813 heartbeat: osd_stat (538 MB used, 3723 GB avail, 3724 GB total Peers [] / [] op hist [] 2016-04-01 1111 peers [] / [] 7f9231e86700 5 osd.139 12813 tick2016-04-01 11VlV 46VO 25.200644 7f9231e86700 10 osd.139 12813 do_waiters-- start2016-04-01 11LV 46RV 25.200648 7f9231e86700 10 osd.139 12813 do_waiters-finish2016-04-01 11RV 46RV 25.600974 7f9219d15700 20 osd.139 12813 update_osd_stat osd_stat (538 MB used, 3723 GB avail, 3724 GB total Peers [] / [] op hist [] (2016-04-01 11) 46 7f9219d15700 25.601002 7f9219d15700 5 osd.139 12813 heartbeat: osd_stat (538 MB used, 3723 GB avail, 3724 GB total Peers [] / [] op hist [] 2016-0401 111411 peers [] / [] op hist [] 2016-0401 11140759 7f9231e86700 5 osd.139 12813 tick2016-0401 1114VOV 26.200784 7f9231e86700 10 osd.139 12813 do_waiters-- start2016-04-01 11VOLV 26.200788 7f9231e86700 10 osd.139 12813 do_waiters-- finish2016-04-01 1114Vol 27.200867 7f9231e86700 5 osd.139 12813 tick2016-04-0111Vol 27.200892 7f9231e86700 10 osd.139 12813 do_waiters-start2016-04-04-11 01 11VV 46 do_waiters 27.200895 7f9231e86700 10 osd.139 12813 do_waiters-- finish2016-04-01 11 Vol 46 Vol 28.201002 7f9231e86700 5 osd.139 12813 tick2016-04-01 11VV 28.201022 7f9231e86700 10 osd.139 12813 do_waiters-- start2016-04-01 11VV 46VV 28.201030 7f9231e86700 10 osd.139 12813 do_waiters-finish2016-04-01 11VV 46VV 29.101147 7f9219d15700 20 osd.139 12813 update_osd_stat osd_stat (538 MB used 3723 GB avail, 3724 GB total, peers [] / [] op hist [] 2016-04-01 11 peers 46 op hist 29.101180 7f9219d15700 5 osd.139 12813 heartbeat: osd_stat (538 MB used, 3723 GB avail, 3724 GB total Peers [] / [] op hist [] 2016-04-0111: 46 osd.139 29.201115 7f9231e86700 5 osd.139 12813 tick2016-04-0111: 46 osd.139 29.201128 7f9231e86700 10 osd.139 12813 do_waiters-- start2016-04-0111: 46 osd.139 29.201132 7f9231e86700 10 osd.139 12813 do_waiters-- finish2016-04-0111: 4630.201237 7f9231e86700 5 osd.139 12813 tick2016-04-0111: 4630 201267 7f9231e86700 10 osd.139 12813 do_waiters-- start2016-04-04-0111 01 111VOLGUL 46VOV 30.201271 7f9231e86700 10 osd.139 12813 do_waiters-- finish

# solution 1. Delete the corresponding osd information in crush

Ceph osd crush remove osd.139 # Note that it may lead to data migration

two。 Start the osd service and add the osd back into the crushmap.

Ceph osd crush add 139 1.0 host=xxx after reading the above, do you have any further understanding of what to do if osd cannot up due to the frequent addition and deletion of osd? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.