CEPH OSD version mis-match

[barryyancey]

I am running pve-manager/8.2.7/3e0176e6bb2ade3b with CEPH Reef. I have 2 nodes with OSDs at 18.2.2 and 1 node at 18.2.4. I see the upgrade icons on the CEPH screen that say to upgrade to the latest version, but I cannot find out how to accomplish that. I have enterprise subscriptions for all three nodes. I have run the upgrade refresh multiple times, verified that the repository is correct, and done everything short of rebuilding the CEPH install. The 2 nodes that are 18.2.2 are older nodes that were part of the cluster for months. I just added the 3rd one to achieve HA and that one has the 18.2.4. Nothing is different though about that box. They are all running the same software and virtually the same hardware. Not sure what I am missing.

root@ares:~# ceph -s
cluster:
id: c7dfaacf-cc78-45ad-b82b-22616b3b2901
health: HEALTH_OK

services:
mon: 3 daemons, quorum apollo,ares,artemis (age 11m)
mgr: apollo(active, since 24h), standbys: artemis, ares
mds: 1/1 daemons up, 2 standby
osd: 12 osds: 12 up (since 10m), 12 in (since 56m)

data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 20.92k objects, 80 GiB
usage: 240 GiB used, 15 TiB / 15 TiB avail
pgs: 97 active+clean

io:
client: 179 KiB/s wr, 0 op/s rd, 19 op/s wr

root@ares:~# ceph versions
{
"mon": {
"ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)": 2,
"ceph version 18.2.4 (2064df84afc61c7e63928121bfdd74c59453c893) reef (stable)": 1
},
"mgr": {
"ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)": 2,
"ceph version 18.2.4 (2064df84afc61c7e63928121bfdd74c59453c893) reef (stable)": 1
},
"osd": {
"ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)": 8,
"ceph version 18.2.4 (2064df84afc61c7e63928121bfdd74c59453c893) reef (stable)": 4
},
"mds": {
"ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)": 2,
"ceph version 18.2.4 (2064df84afc61c7e63928121bfdd74c59453c893) reef (stable)": 1
},
"overall": {
"ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)": 14,
"ceph version 18.2.4 (2064df84afc61c7e63928121bfdd74c59453c893) reef (stable)": 7
}
}

root@ares:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 15.28271 - 15 TiB 240 GiB 233 GiB 186 KiB 6.7 GiB 15 TiB 1.53 1.00 - root default
-3 5.82196 - 5.8 TiB 83 GiB 78 GiB 84 KiB 5.1 GiB 5.7 TiB 1.39 0.91 - host apollo
8 ssd 1.45549 1.00000 1.5 TiB 16 GiB 15 GiB 29 KiB 1.5 GiB 1.4 TiB 1.10 0.72 22 up osd.8
9 ssd 1.45549 1.00000 1.5 TiB 35 GiB 34 GiB 16 KiB 1.6 GiB 1.4 TiB 2.37 1.55 32 up osd.9
10 ssd 1.45549 1.00000 1.5 TiB 20 GiB 19 GiB 15 KiB 1.2 GiB 1.4 TiB 1.37 0.89 24 up osd.10
11 ssd 1.45549 1.00000 1.5 TiB 11 GiB 9.8 GiB 24 KiB 782 MiB 1.4 TiB 0.71 0.46 19 up osd.11
-7 3.63879 - 3.6 TiB 78 GiB 78 GiB 73 KiB 467 MiB 3.6 TiB 2.10 1.37 - host ares
0 ssd 0.90970 1.00000 932 GiB 12 GiB 12 GiB 3 KiB 70 MiB 919 GiB 1.33 0.86 16 up osd.0
1 ssd 0.90970 1.00000 932 GiB 19 GiB 19 GiB 29 KiB 132 MiB 912 GiB 2.06 1.34 23 up osd.1
2 ssd 0.90970 1.00000 932 GiB 17 GiB 17 GiB 24 KiB 121 MiB 914 GiB 1.84 1.20 24 up osd.2
3 ssd 0.90970 1.00000 932 GiB 30 GiB 29 GiB 17 KiB 144 MiB 902 GiB 3.17 2.07 34 up osd.3
-5 5.82196 - 5.8 TiB 79 GiB 78 GiB 29 KiB 1.2 GiB 5.7 TiB 1.32 0.86 - host artemis
4 ssd 1.45549 1.00000 1.5 TiB 27 GiB 27 GiB 10 KiB 289 MiB 1.4 TiB 1.81 1.18 27 up osd.4
5 ssd 1.45549 1.00000 1.5 TiB 17 GiB 17 GiB 6 KiB 172 MiB 1.4 TiB 1.16 0.76 22 up osd.5
6 ssd 1.45549 1.00000 1.5 TiB 7.5 GiB 7.2 GiB 5 KiB 228 MiB 1.4 TiB 0.50 0.33 17 up osd.6
7 ssd 1.45549 1.00000 1.5 TiB 27 GiB 27 GiB 8 KiB 498 MiB 1.4 TiB 1.82 1.18 31 up osd.7
TOTAL 15 TiB 240 GiB 233 GiB 190 KiB 6.7 GiB 15 TiB 1.53
MIN/MAX VAR: 0.33/2.07 STDDEV: 0.71

 

[VoIP-Ninja]

Hi there,

looks like "Artemis" is up-to-date because is it the latest node added, with the latest version.

• Update both other nodes ( package update ).
• Restart your monitor nodes, one by one on the 2 "older" nodes.
• Restart all MDS if you run CephFS), one by one.
• Restart your OSDs one by one

 

[barryyancey]

I had done all of this but it still did not pick up the new version for some reason. After about a week or so, the updates finally showed up and then I applied them. I am just not sure why it took so long for them to be available on the other nodes. They are all running the same repos with the same licensing, on the same account. Very odd. Thank you for your response though.