Shulou(Shulou.com)05/31 Report--

This article shares with you the content of the sample analysis of Ceph heartbeat and the Internet. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

An unhealthy state of Ceph

Environment: three machines bdc212,bdc213,bdc213, each machine two osd, three copies, and each machine is a dual network, respectively, 192.168.8.0 network segment ip and 192.168.12.0 network segment.

After the installation has just been completed, the test cluster is restarted. After the osd restart of bdc212, it is found that the cluster status has been ERR, as shown below:

# ceph-s cluster befd161e-ce9e-427d-9d9b-b683b744645c health HEALTH_ERR 60 pgs are stuck inactive for more than 300 seconds 256 pgs peering 60 pgs stuck inactive monmap e3: 3 mons at {bdc212=192.168.13.212:6789/0,bdc213=192.168.13.213:6789/0,bdc214=192.168.13.214:6789/0} election epoch 48, quorum 0up 1 bdc212,bdc213,bdc214 osdmap e572: 6 osds: 6 up, 6 in 92 remapped pgs flags sortbitwise pgmap v378651: 256 pgs, 2 pools, 7410 MB data, 1 objects 22517 MB used, 22288 GB / 22310 GB avail 164 peering 92 remapped+peering

Check the log and find that when there is an ip address mismatch, it is obvious that the two ip addresses do not match in the red part. Restart each machine many times and find that sometimes it can match, and the cluster returns to normal, and most of the time it is an error.

2016-06-06 17 INF 4949 osd.1 11.262293 mon.0 192.168.13.212 12V 6789Bank 0294: cluster [INF] osd.1 192.168.13.212 12V 6804max 277426 boot2016-06-06 174949426 boot2016-06-06 174949 osd.1 11.2620 mon.0 192.168.13.212purl 6789295: cluster [INF] osd.0 192.168.13.212purl 6800Z277433 boot2016-06-06 17RM 4911.264871 mon.0 192.168.13.212rel 67890296: Cluster [INF] osdmap e570: 6 osds: 6 up 6 in2016-06-06 17 cluster 49 INF 49 INF 11.267704 mon.0 192.168.13.212 12 Swiss 6789 bank 0297: cluster [INF] pgmap v378644: 256 pgs: 48 stale+active+clean, 126 peering, 82 active+clean 7410 MB data, 22514 MB used, 22288 GB / 22310 GB avail2016-06-06 1749 GB avail2016 12.272648 mon.0 192.168.13.212 in2016 0298: cluster [INF] osdmap e571: 6 osds: 6 up, 6 in2016-06-06 1749 purge 12.282714 mon.0 192.168.13.212 purge 6789Univer 0299: cluster [INF] pgmap v378645: 256 pgs: 48 stale+active+clean, 208 peering 7410 MB data, 22516 MB used, 22288 GB / 22310 GB avail2016-06-06 1749 mon.0 192.168.13.212 mon.0 192.168.13.212 osds 0300: cluster [INF] osdmap e572: 6 osds: 6 up, 6 in2016-06-06 17purge 13.275671 mon.0 192.168.13.212 pgmap 6789Univer 0301: cluster [INF] pgmap v378646: 256 pgs: 48 stale+active+clean, 208 peering 7410 MB data, 22516 MB used 22288 GB / 22310 GB avail2016-06-06 17osd.1 192.168.13.212 osd.1 6804ax 277426 1: cluster [ERR] map e571 had wrong cluster addr (192.168.13.212 osd.1 6806806806277426! = my 192.168.212purl 6806806277426) 2016-06 17Frange 4912.268718 osd.0 192.168.13.21218186800277433 1: cluster [ERR] map e571 had wrong cluster addr (192.168.13.212A 6801, 277433) ! = my 192.168.8.212 cluster 6801 pgs 277433) 2016-06-06 17 cluster 4949 14.303244 mon.0 192.168.13.212 cluster [INF] pgmap v378647: 256 remapped+peering 48 stale+active+clean, 116 peering 7410 MB data, 22518 MB used, 22288 GB / 22310 GB avail2016-06-06 1749 GB avail2016 16.686779 mon.0 192.168.13.212 GB 0303: cluster [INF] pgmap v378648: 256pgs: 92 remapped+peering, 48 stale+active+clean, 116 peering; 7410 MB data, 22519 MB used, 22288 GB / 22310 GB avail2016-06-06 17,917.693779 mon.0 192.168.13.212 Suzhou 6789304: cluster [INF] pgmap v378649: 256 pgs: 92 remapped+peering, 164 peering 7410 MB data, 22517 MB used, 22288 GB / 22310 GB avail2016-06-06 174914 mon.0 192.168.13.212 mon.0 192.168.13.212 GB 0305: cluster [INF] pgmap v378650: 256 pgs: 92 remapped+peering, 164164 peering; 7410 MB data, 22517 MB used, 22288 GB / 222310 GB avail2016-06-06 17494914916720 mon.0 192.168.13.212 12V378651: 256peering: cluster [INF] pgmap v378651: 256 peering 7410 MB data, 22517 MB used, 22288 GB / 22310 GB avail2016-06-06 1749 GB avail2016 24.256323 mon.0 192.168.13.212 mon.0 0307: cluster [INF] HEALTH_ERR; 60 pgs are stuck inactive for more than 300 seconds; 256 pgs peering 60 pgs stuck inactive2016-06-06 17 cluster 49 INF 55.276736 mon.0 192.168.13.212 12 cluster [INF] osd.0 192.168.13.212 12V 6800 failed (2 reporters from different host after 20.648970 > = grace 20.000000) 2016-06-06 1749 V 49V 55.276847 mon.0 192.168.13.212 12V 6789323: cluster [INF] osd.1 192.168.13.212 12 purl 680426 failed (2 reporters from different host after 20.648910 > = grace 20.000000)

Cluster [ERR] map e571 had wrong cluster addr (192.168.13.212 map 6806 Universe 277426! = my 192.168.212)

Cluster [ERR] map e571 had wrong cluster addr (192.168.13.212 map 6801 Universe 277433! = my 192.168.212 map 6801 Universe 277433)

Refer to the configuration of the official website and conduct the following three tests:

First test: add heartbeat address and host

Modify the configuration file

[osd.0]

Host = bdc212

Osd heartbeat address = 192.168.8.212

[osd.1]

Host = bdc212

Osd heartbeat address = 192.168.8.212

Restart the osd of bdc212:

# systemctl restart ceph-osd.target

Check ceph-w and find that after restart, the configuration does not work. In addition, the heartbeat address is changed to 192.168.13.212, and the cluster status is still ERR.

2016-06-06 10 remapped+peering; 35 mon.0 43.828677 mon.0 [INF] osdmap e540: 6 osds: 4 up, 6 in2016-06-06 10 osdmap 35 remapped+peering; 375 786 mon.0 [INF] v378545: 256 pgs: 26 peering, 230 remapped+peering; 7410 MB data, 22505 MB used, 22288 GB / 22310 GB avail2016-06-06 10 10 peering 10 10 pgmap v378546: 256126 active+undersized+degraded, 26 peering, 104 remapped+peering; 7410 MB data, 22506 MB used, 22288 GB / 22310 GB avail 1Xue 3 objects degraded (33.333%) 2016-06-06 10 mon.0 35 pgs 47.878381 mon.0 [INF] pgmap v378547: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22507 MB used, 22288 GB / 22310 GB avail; 1 GB avail; 3 objects degraded (33.333%) 2016-06-06 10 pgmap 50.365666 mon.0 [INF] pgmap v378548: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail 52.817568 mon.0 [INF] pgmap v378549: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail; 1 objects degraded (33.333%) 2016-06-06 pgs 1035 mon.0 53.820444 mon.0 [INF] pgmap v378550: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail 1 INF 3 objects degraded (33.333%) 2016-06-06 10 INF 36 INF 24.144881 mon.0 [INF] HEALTH_WARN; 256 pgs degraded; 256 pgs stuck unclean; 256 pgs undersized; recovery 1 Plus 3 objects degraded (33.333%) 2Accord 6 in osds are down2016-06-06 10 INF mon.0 [INF] from='client.? 192.168.13.212 from='client.? 819457382 'entity='osd.1' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 1 "weight": 3.6313}]: dispatch2016-06-06 10 mon.0 39 INF 20.484124 mon.0 [INF] from='client.? 192.168.13.212 from='client.? 2428259865' entity='osd.0' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 0 "weight": 3.6313}]: dispatch2016-06-06 10 mon.0 21.520292 mon.0 [INF] osd.1 192.168.13.212 INF 6800MB 25527679 boot2016-06-06 10 mon.0 21.520479 mon.0 [INF] osd.0 192.168.13.212 osd.0 192.168.13.212 boot2016-06-06 10 mon.0 3915 mon.0 [INF] osdmap e541: 6 osds: 6 up, 6 in2016-06-06 10 mon.0 39anger 21.524495 mon.0 [INF] pgmap v378551: 256 pgs: 256 active+undersized+degraded 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail; 1 objects degraded (33.333%) 2016-06-06 10 in2016 3922. 53 1697 mon.0 [INF] osdmap e542: 6 osds: 6 up, 6 in2016-06-06 10 displacement 3922.533615 mon.0 [INF] pgmap v378552: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 2208 MB used, 22288 GB / 22310 GB avail 1Xue 3 objects degraded (33.333%) 2016-06-06 10 INF 39 pgmap 23.549374 mon.0 [INF] pgmap v378553: 256 pgs: 130 remapped+peering, 126 active+undersized+degraded; 7410 MB data, 22509 MB used, 22288 GB / 22310 GB avail; 1 GB avail; 3 objects degraded (33.333%) 2016-06-06 10 39 Swiss Vane 24.145709 mon.0 [INF] HEALTH_WARN; 126 pgs degraded; 130 pgs peering; pgs stuck unclean; 126 pgs undersized Recovery 1Plus 3 objects degraded (33.333%) 2016-06-06 10 mon.0 3915 25.654043 mon.0 [INF] pgmap v378554: 256 pgs: 256 remapped+peering; 7410 MB data, 22509 MB used, 22288 GB / 22310 GB avail2016-06-06 10 GB avail2016 39 mon.0 [INF] pgmap v378555: 256 pgs: 256 remapped+peering; 7410 MB data, 2209 MB used, 22288 GB / 222310 GB avail2016-06-06 10 GB 3928.670168 mon.0 [INF] pgmap v378556: 256 remapped+peering 7410 MB data, 22510 MB used, 22288 GB / 22310 GB avail2016-06-06 10 GB 39 GB 30.678376 mon.0 [INF] pgmap v378557: 256 pgs: 256 remapped+peering; 7410 MB data, 222510 MB used, 22288 GB / 22310 GB avail2016-06-06 10VV 3932.687783 mon.0 [INF] pgmap v378558: 256 pgs: 256 remapped+peering; 7410 MB data, 22509 MB used, 22288 GB / 22310 GB avail2016-06-06 10 GB 39V33.697850 mon.0 [INF] pgmap v378559: 256 pgs: 256 remapped+peering 7410 MB data, 22509 MB used 22288 GB / 22310 GB avail2016-06-06 10 osd.1 3922 osd.1 [ERR] map e542 had wrong cluster addr (192.168.13.212 12 map 255276! = my 192.168.212 12 map 6801 Great 255276) 2016-06-06 10 osd.1 3951.748896 osd.0 [ERR] map e542 had wrong cluster addr (192.168.13.212 12 had wrong cluster addr 6805 hand 255283! = my 192.168.212 1218 6805283) .0 [INF] osd.0 192.168.13.212 failed 6804 255283 failed (2 reporters from different host after 21.000496 > = grace 20.000000) 2016-06-06 10 failed 40 mon.0 10.350839 mon.0 [INF] osd.1 192.168.13.212 18800 failed (2 reporters from different host after 21.001158 > = grace 20.000000) 2016-06 10 40 mon.0 [INF] osdmap e543: 6 osds: 4 up 6 in2016-06-06 10 pgs 40 INF 10.419756 mon.0 [INF] pgmap v378560: 256 pgs: 256 remapped+peering 7410 MB data, 22509 MB used, 22288 GB / 22310 GB avail2016-06-06 10 GB 4040 mon.0 11.416130 mon.0 [INF] osdmap e544: 6 osds: 4 up, 6 in2016-06-06 1040 purge 11.418453 mon.0 [INF] pgmap v378561: 256 pgs: 256 remapped+peering; 7410 MB data, 22509 MB used, 22288 GB / 22310 GB avail2016-06-06 10140 40mer 13.446063 mon.0 [INF] pgmap v378562 pgs: 130active+undersized+degraded, 126 remapped+peering 7410 MB data, 22510 MB used, 22288 GB / 22310 GB avail second test: add static ip

Modify the configuration file

[osd.0]

Cluster addr = 192.168.8.212

[osd.1]

Cluster addr = 192.168.8.212

Restart bdc212's osd service again, and the test shows that static ip works not only for 8 network segments, but also for 13 network segments.

2016-06-06 10 HEALTH_WARN; 45 pgs degraded; 24.147360 mon.0 [INF] HEALTH_WARN; 160 pgs degraded; 256 pgs stuck unclean; 160 pgs undersized; recovery 1 pgs undersized; recovery 3 objects degraded (33.333%) 2016-06-06 10 10 HEALTH_WARN; 45 active+remapped 25.924692 mon.0 [INF] pgmap v378575: 256 pgs: 96 active+remapped, 160 active+undersized+degraded; 7410 MB data, 15015 MB used, 14859 GB / 14873 GB avail 1Xue 3 objects degraded (33.333%) 2016-06-06 10 INF 45 pgmap 27.932786 mon.0 [INF] pgmap v378576: 256 pgs: 96 active+remapped, 160 active+undersized+degraded; 7410 MB data, 15015 MB used, 14859 GB / 14873 GB avail 1ax 3 objects degraded (33.333%) 2016-06-06 10 INF 52 from='client.? 48.811978 mon.0 [INF] from='client.? 192.168.13.212 entity='osd.1' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 1 "weight": 3.6313}]: dispatch2016-06-06 10 mon.0 48.813694 mon.0 [INF] from='client.? 192.168.13.212 from='client.? 4288153588 'entity='osd.0' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 0 "weight": 3.6313}]: dispatch2016-06-06 10 mon.0 52purl 49.881762 mon.0 [INF] osd.1 192.168.13.212 osd.1 192.168.13.212 boot2016-06-06 10VOV 52purl 49.882013 mon.0 [INF] osd.0 192.168.13.2121800 261514 boot2016-06-06 10purl 52purl 49.884250 mon.0 [INF] osdmap e548: 6 osds: 6 up, 6 in2016-06-06 102purl 52purl 49.886696 mon.0 [INF] pgmap v378577: pgs: 96 active+remapped 160 active+undersized+degraded 7410 MB data, 15015 MB used, 14859 GB / 14873 GB avail; 1 objects degraded (33.333%) 2016-06-06 10 objects degraded 52GB 50.904273 mon.0 [INF] osdmap e549: 6 osds: 6 up, 6 in2016-06-06 10 in2016 52V378579: 256 pgs: 126remapped+peering, 51 active+remapped, 79 active+undersized+degraded 7410 MB data, 15015 MB used, 14859 GB / 14873 GB avail2016-06-0610: 52 GB avail2016 51.906771 mon.0 [INF] osdmap e550: 6 osds: 6 up, 6 in2016-06-0610: 52 GB 51.909719 mon.0 [INF] pgmap v378580: 256 pgs: 126remapped+peering, 51 active+remapped, 79 active+undersized+degraded; 7410 MB data, 15015 MB used, 14859 GB / 14873 GB avail2016-06-0610: 52mer 53.920610 mon.0 [INF] pgmap v378581: 256pgs: 82 active+clean, 126 remapped+peering, 35 active+remapped, 13 active+undersized+degraded 7410 MB data, 15016 MB used, 14859 GB / 14873 GB avail2016-06-06 10 mon.0 55.941817 mon.0 [INF] pgmap v378582: 256 pgs: 256 active+clean; 7410 MB data, 222518 MB used, 22288 GB / 22310 GB avail2016-06-06 10 mon.0 52GB 58.057115 mon.0 [INF] pgmap v378583: 256 pgs: 256 active+clean; 7410 MB data, 222519 MB used, 22288 GB / 22310 GB avail2016-06-06 1053 GB 00.065731 mon.0 [INF] pgmap 378584: 256 pgs: 256 active+clean 7410 MB data, 22519 MB used, 22288 GB / 22310 GB avail2016-06-06 1053 MB used 01.069834 mon.0 [INF] pgmap v378585: 256pgs: 256active+clean; 7410 MB data, 222519 MB used, 22288 GB / 22310 GB avail2016-06-06 1053 GB 03.085171 mon.0 [INF] pgmap v378586: 256 pgs: 256 active+clean; 7410 MB data, 22519 MB used, 22288 GB / 22310 GB avail2016-06-06 10 GB 53 MB data, 222519 MB used, 22288 GB / 22310 GB avail2016-06-06 10V 53 mon.0 [INF] pgmap v378586: 256 pgs: 256 active+clean; 7410 MB data, 22519 MB used, 22288 GB / 22310 GB avail2016-06-06 1015 GB

It was found that the cluster returned to normal.

The third test: add cluster network

First simulate the ERR state that appeared before the first test, and then

Add: under the global section:

Cluster network = 192.168.13.212Accord16

Restart again:

2016-06-06 17 3414 24 251564 mon.0 [INF] HEALTH_WARN 256 pgs peering2016-06-06 17 pgs peering2016 34 ERR 37.582278 osd.1 [ERR] map e562 had wrong cluster addr (192.168.13.212 ERR 6805 peg 276029! = my 192.168.212 map 6805 peg 276029! = 192.168.212 192.168.212 map e562 had wrong cluster addr (192.168.13.212 ERR 6801 hand 276025! = my 192.168.212 12 my 6801 mon.0 276025) 2016-06-06 17 34 mon.0 56.509186 mon.0 INF] osd.0 192.168.13.212 failed 6800 failed (2 reporters from different host after 22.645655 > = grace 20.000000) 2016-06-06 17 failed 56.509895 mon.0 [INF] osd.1 192.168.13.212 12 mon.0 6804 failed 276029 (2 reporters from different host after 22.646360 > = grace 20.000000) 2016-06 171404 mon.0 [INF] osdmap e563: 6 osds: 4 up 6 in2016-06-06 17 INF 34 pgmap 56.576604 mon.0 [INF] pgmap v378626: 256 pgs: 256 remapped+peering 7410 MB data, 22505 MB used, 22288 GB / 22310 GB avail2016-06-06 17mon.0 57.584605 mon.0 [INF] osdmap e564: 6 osds: 4 up, 6 in2016-06-06 17mon.0 34INF 57.589648 mon.0 [INF] pgmap v378627: 255pgs: 256 remapped+peering; 7410 MB data, 22505 MB used, 22288 GB / 22310 GB avail2016-06-06 171418 mon.0 [INF] pgmap v378628: 256 pgs: 126 active+undersized+degraded, 130 remapped+peering; 7410 MB data, 22505 MB used, 22288 GB / 22310 GB avail 1.623694 mon.0 [INF] pgmap v378629: 256pgs: 256active+undersized+degraded; 7410 MB data, 22507 MB used, 22288 GB / 22310 GB avail; 1 objects degraded (33.333%) 2016-06-06 1735 pgmap 03.919856 mon.0 [INF] pgmap v378630: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail 1 active+undersized+degraded; 3 objects degraded (33.333%) 2016-06-06 17 mon.0 35 mon.0 06.564530 mon.0 [INF] pgmap v378631: 256 pgs: 256 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail; 1 objects degraded (33.333%) 2016-06-06 1735 mon.0 [INF] HEALTH_WARN; 256 pgs degraded; 256 pgs stuck unclean; 256 pgs undersized; recovery 1 objects degraded (33.333%) 2According 6 in osds are down2016-06-06 1714 entity='osd.1' cmd= 36 mon.0 22.468740 mon.0 [INF] from='client.? 192.168.13.212 Bermuda 2865614433 'entity='osd.1' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 1 "weight": 3.6313}]: dispatch2016-06-06 17 mon.0 3622.490094 mon.0 [INF] from='client.? 192.168.13.212 from='client.? 0pee 1154763864' entity='osd.0' cmd= [{"prefix": "osd crush create-or-move", "args": ["host=bdc212", "root=default"], "id": 0 "weight": 3.6313}]: dispatch2016-06-06 17 mon.0 3615 mon.0 23.534519 mon.0 [INF] osd.1 192.168.13.212 1400 276823 boot2016-06-06 1731 276823 boot2016-06-06 1731 2768 23 mon.0 [INF] osd.0 192.168.13.212 12 boot2016-06 17804 276828 boot2016-06-06 171818 3688 mon.0 [INF] osdmap e565: 6 osds: 6 up, 6 in2016-06-06 173V 3623.538601 mon.0 [INF] pgmap v378632: 256 pgs: 256 active+undersized+degraded 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail; 1 pgs undersized; recovery 3 objects degraded (33.333%) 2016-06-06 17 GB 36V24.252318 mon.0 [INF] HEALTH_WARN; 256 pgs degraded; 256 pgs stuck unclean; 256 pgs undersized; recovery 1 pgs undersized; recovery 3 objects degraded (33.333%) 2016-06-06 17VAR 36 objects degraded 24.551431 mon.0 [INF] pgmap v378633: 256 pgs: 126 remapped+peering, 130 active+undersized+degraded 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail2016-06-06 1736 remapped+peering 24.568370 mon.0 [INF] osdmap e566: 6 osds: 6 up, 6 in2016-06-06 1736 mon.0 [INF] pgmap v378634: 256 pgs: 126remapped+peering, 130 active+undersized+degraded 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail2016-06-06 1736 MB data 25.577264 mon.0 [INF] osdmap e567: 6 osds: 6 up, 6 in2016-06-06 17V 3625 581312 mon.0 [INF] pgmap v378635: 256 pgs: 126remapped+peering, 130 active+undersized+degraded; 7410 MB data, 22508 MB used, 22288 GB / 222310 GB avail2016-06-06 1717INF 26.591915 mon.0 [INF] pgmap v378636: 256 pgs: 82 active+clean, 126 remapped+peering, 48 active+undersized+degraded 7410 MB data, 22508 MB used, 22288 GB / 22310 GB avail2016-06-06 171414 mon.0 28.751459 mon.0 [INF] pgmap v378637: 256 pgs: 178active+clean, 78 remapped+peering; 7410 MB data, 22510 MB used, 22288 GB / 22310 GB avail2016-06-06 17virtual 3629 active+clean; 758035 mon.0 [INF] pgmap v378638: 256 pgs: 256 active+clean; 7410 MB data, 22511 MB used, 22288 GB / 22310 GB avail2016-06-06 1717V31.774843 mon.0 [INF] pgmap v378639: 256 pgs: 256 active+clean 7410 MB data, 22513 MB used, 22288 GB / 22310 GB avail2016-06-06 17MB data 33783225 mon.0 [INF] pgmap v378640: 256 pgs: 256 active+clean; 7410 MB data, 222513 MB used, 22288 GB / 22310 GB avail2016-06-06 1717MB data 34.786234 mon.0 [INF] pgmap v378641: 256 pgs: 256 active+clean; 7410 MB data, 222514 MB used, 22288 GB / 22310 GB avail2016-06-06 17purl 37 24252649 mon.0 [INF] HEALTH_OK

It was found that the cluster also returned to normal.

Although the cluster has returned to normal, the theoretical knowledge is not very clear. Continue to learn the heartbeat mechanism and the configuration of Ceph network from the official website:

Http://docs.ceph.org.cn/rados/configuration/mon-osd-interaction/#index-6

Http://docs.ceph.org.cn/rados/configuration/network-config-ref/

The test a year later

Public network

To configure a public network, add the following options to the [global] section of the configuration file.

[global]... Public network = {public-network/netmask}

Trunking network

If you declare a trunked network, OSD will route heartbeat, object replication, and recovery traffic to the clustered network, which will improve performance compared to a single network. To configure a cluster, add the following options to the [global] section of the configuration file.

[global]... Cluster network = {cluster-network/netmask}

Did a little test.

Configure in / etc/ceph/ceph.conf

Cluster_network=192.16.40.0/24

Public_network=172.16.40.0/24

If you restart ceph, you will find that both the public network and the cluster network are configured. The osd network will give priority to the use of the public network. The log shows

2017-06-13 16 INF 14 mon.0 54.305773 192.16.40.1 mon.0 6789 boot 0 23: cluster [INF] osd.2 172.16.40.1 18800 boot

Remove public_network and configure only cluster network. Osd will use cluster network.

2017-06-13 16 INF 2343.979744 mon.0 192.16.40.1 INF 6789 boot 0 33: cluster [INF] osd.2 192.16.40.1 18817 boot

Thank you for reading! This is the end of this article on "sample Analysis of Ceph heartbeat and Network". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.