CEPH: outdated OSDs after minor upgrade

[Lephisto]

Hi,

i just ran into an issue after updating PVE / Ceph today:

The Ceph Packages were upgraded from 14.2.4 to 14.2.4.1. Everything works, pool is healthy, just the UI is showing "outdated OSDs", because the Ceph Nodes still thing they're 14.2.4 but the OSDs are 14.2.4.1:



What do I miss here?

thanks in advance.

 

[Mike Lupe]

Hi - imo either chose each OSD in a row and press "Restart" (for each one), or reboot the nodes. Don't forget to take the usual precautions

 

[Lephisto]

Hi Mike,

thanks for answering.

The problem is not the outdated OSDs but a Version mismatch between Host and OSD. OSDs are newer and don't match what Proxmox accounts for the host. Reboot doesn't do anything (hosts have been rebooted after ceph upgrade anyways).

Hi - imo either chose each OSD in a row and press "Restart" (for each one), or reboot the nodes. Don't forget to take the usual precautions

 

[tom]

if you got new ceph packages via apt, do the following.

1. restart all Ceph monitors, node by node
2. restart all Ceph MDS, node by node (if you run CephFS)
3. restart all OSD, one by one
4. you can do all this via GUI
   (if you know what you are doing you can restart all OSD on a node with 'systemctl restart ceph-osd.target', noder per node, make sure that the cluster is healthy again before you do the same on the next node)

the new GUI shows if you have newer packages installed but the service is still running the old binary.

 

[Lephisto]

I booted several times now, restarted monitors (no mds since i don't have cephfs), restarted osd's, restarted ceph-osd.target on all nodes. Still the same.

//edit: btw i checked with dpkg, all ceph* packages are on 14.2.4.1

 

[tom]

maybe a browser caching issue?

 

[Lephisto]

Nope. Same on several Workstations / Sessions.

 

[Lephisto]

Maybe this bump hasn't made it into all packages?

>> https://code.forksand.com/proxmox/ceph/src/branch/master/patches/0014-bump-version-to-14.2.4.1.patch

 

[jbukowskipl]

i have the same problem after apt-get upgrade today..

 

[tom]

i have the same problem after apt-get upgrade today..

never run apt-get upgrade.

read the general upgrade guide for Proxmox VE.

 

[Romsch]

i have the same problem after apt-get upgrade today..

Maybe this bump hasn't made it into all packages?

>> https://code.forksand.com/proxmox/ceph/src/branch/master/patches/0014-bump-version-to-14.2.4.1.patch

I had the same - if you have no subcription. Check the sources.list, if it looks like this; it should work then:

deb http://ftp.debian.org/debian buster main contrib
deb http://ftp.debian.org/debian buster-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

# security updates
deb http://security.debian.org/debian-security buster/updates main contrib





Best regards, roman

 

[starkruzr]

never run apt-get upgrade.

read the general upgrade guide for Proxmox VE.

Huh? Unless you're saying "always do dist-upgrade."

 

[Lephisto]

I ran into the same issue some time ago on a Lab system. Only do a dist-upgrade procedure, because it checks dependencies in a smarter way. Especially the ceph components can be pretty fucked up that way..

 

[starkruzr]

I thought the difference between upgrade and dist-upgrade was that vanilla upgrade wouldn't upgrade kernels, and that was about it.

 

[jameshost]

Hey sorry to bump this years-old thread but I just ran into this same issue and the fix was just to destroy and re-create the monitor on the failing hosts. Didn't see anyone say this above, and somehow the race condition still exists where on a OSD fleet upgrade, some nodes will come back with this version mismatch.

 

[acme64]

i've got the same issue. but i don't have any failing monitors, they're all updated. ah i had to reboot the nodes from the osd section as well as monitor.