Hello
Status before upgrade:
cluster of 7 server's running 3.4-11 with ceph hammer.
status now:
ceph -
root@pm05:~# ceph -s
cluster 2c01ea87-4de7-4941-9c0a-56b6dd14410a
health HEALTH_WARN
42 pgs backfill
3 pgs backfill_toofull
18 pgs backfilling
1 pgs degraded
1 pgs stuck degraded
60 pgs stuck unclean
1 pgs stuck undersized
1 pgs undersized
recovery 2200/1047748 objects degraded (0.210%)
recovery 277877/1047748 objects misplaced (26.521%)
monmap e6: 4 mons at {0=10.102.3.1:6789/0,1=10.102.3.2:6789/0,2=10.102.3.3:6789/0,3=10.102.3.7:6789/0}
election epoch 3564, quorum 0,1,2,3 0,1,2,3
osdmap e7853: 18 osds: 18 up, 18 in; 60 remapped pgs
pgmap v11101486: 224 pgs, 2 pools, 1141 GB data, 288 kobjects
3752 GB used, 14535 GB / 18287 GB avail
2200/1047748 objects degraded (0.210%)
277877/1047748 objects misplaced (26.521%)
163 active+clean
39 active+remapped+wait_backfill
17 active+remapped+backfilling
3 active+remapped+wait_backfill+backfill_toofull
1 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfilling
recovery io 61219 kB/s, 14 objects/s
client io 7541 kB/s rd, 4984 kB/s wr, 752 op/s
while upgrading the cluster, we lost some of the OSD and couldn't bring them back up, now we have two servers that belong to old cluster:
Version: 6.2.0
Config Version: 13
Cluster Name: BezeqINT01
Cluster Id: 22081
Cluster Member: Yes
Cluster Generation: 1924
Membership state: Cluster-Member
Nodes: 2
Expected votes: 7
Total votes: 2
Node votes: 1
Quorum: 4 Activity blocked
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: pm05
Node ID: 5
Multicast addresses: 239.192.86.151
Node addresses: 10.100.3.5
and 4 servers in new cluster:
Quorum information
------------------
Date: Mon Feb 22 10:52:15 2016
Quorum provider: corosync_votequorum
Nodes: 4
Node ID: 0x00000005
Ring ID: 2680
Quorate: Yes
Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 4
Quorum: 3
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.100.3.1
0x00000002 1 10.100.3.2
0x00000005 1 10.100.3.3 (local)
0x00000003 1 10.100.3.7
after rbd came back to warn from err we started booting up vms and found that one(!) server is missing, we can see it from the old cluster but it does not exist on new one, included screenshots.
Status before upgrade:
cluster of 7 server's running 3.4-11 with ceph hammer.
status now:
ceph -
root@pm05:~# ceph -s
cluster 2c01ea87-4de7-4941-9c0a-56b6dd14410a
health HEALTH_WARN
42 pgs backfill
3 pgs backfill_toofull
18 pgs backfilling
1 pgs degraded
1 pgs stuck degraded
60 pgs stuck unclean
1 pgs stuck undersized
1 pgs undersized
recovery 2200/1047748 objects degraded (0.210%)
recovery 277877/1047748 objects misplaced (26.521%)
monmap e6: 4 mons at {0=10.102.3.1:6789/0,1=10.102.3.2:6789/0,2=10.102.3.3:6789/0,3=10.102.3.7:6789/0}
election epoch 3564, quorum 0,1,2,3 0,1,2,3
osdmap e7853: 18 osds: 18 up, 18 in; 60 remapped pgs
pgmap v11101486: 224 pgs, 2 pools, 1141 GB data, 288 kobjects
3752 GB used, 14535 GB / 18287 GB avail
2200/1047748 objects degraded (0.210%)
277877/1047748 objects misplaced (26.521%)
163 active+clean
39 active+remapped+wait_backfill
17 active+remapped+backfilling
3 active+remapped+wait_backfill+backfill_toofull
1 active+clean+scrubbing+deep
1 active+undersized+degraded+remapped+backfilling
recovery io 61219 kB/s, 14 objects/s
client io 7541 kB/s rd, 4984 kB/s wr, 752 op/s
while upgrading the cluster, we lost some of the OSD and couldn't bring them back up, now we have two servers that belong to old cluster:
Version: 6.2.0
Config Version: 13
Cluster Name: BezeqINT01
Cluster Id: 22081
Cluster Member: Yes
Cluster Generation: 1924
Membership state: Cluster-Member
Nodes: 2
Expected votes: 7
Total votes: 2
Node votes: 1
Quorum: 4 Activity blocked
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: pm05
Node ID: 5
Multicast addresses: 239.192.86.151
Node addresses: 10.100.3.5
and 4 servers in new cluster:
Quorum information
------------------
Date: Mon Feb 22 10:52:15 2016
Quorum provider: corosync_votequorum
Nodes: 4
Node ID: 0x00000005
Ring ID: 2680
Quorate: Yes
Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 4
Quorum: 3
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.100.3.1
0x00000002 1 10.100.3.2
0x00000005 1 10.100.3.3 (local)
0x00000003 1 10.100.3.7
after rbd came back to warn from err we started booting up vms and found that one(!) server is missing, we can see it from the old cluster but it does not exist on new one, included screenshots.