Ceph Outdated OSD's even though on 16.2.6

[jasonsansone]

I have two different clusters prompting that all ODS's, monitors, managers, and MDS servers need to be updated even though they are on the newest Ceph Pacific 16.2.6.

Spoiler: Packages

proxmox-ve: 7.0-2 (running kernel: 5.11.22-7-pve) pve-manager: 7.0-14 (running version: 7.0-14/a9dbe7e3) pve-kernel-helper: 7.1-4 pve-kernel-5.11: 7.0-10 pve-kernel-5.11.22-7-pve: 5.11.22-12 ceph: 16.2.6-pve2 ceph-fuse: 16.2.6-pve2 corosync: 3.1.5-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: not correctly installed ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.22-pve1 libproxmox-acme-perl: 1.4.0 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.0-6 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-12 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-3 libpve-storage-perl: 7.0-13 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-4 lxcfs: 4.0.8-pve2 novnc-pve: 1.2.0-3 proxmox-backup-client: 2.0.13-1 proxmox-backup-file-restore: 2.0.13-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.3-6 pve-cluster: 7.0-3 pve-container: 4.1-1 pve-docs: 7.0-5 pve-edk2-firmware: 3.20210831-1 pve-firewall: 4.2-5 pve-firmware: 3.3-3 pve-ha-manager: 3.3-1 pve-i18n: 2.5-1 pve-qemu-kvm: 6.1.0-1 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-18 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.1.1-pve1

 

[tom]

Did you restart your services after you upgraded the packages?

In short:

1. restart Monitors on all nodes
2. restart Manager on all nodes
3. restart Meta Data Server
4. restart OSD (step by step, make sure that your ceph status is green again before you restart the next OSD

 

[dcsapak]

AFAICS, this is currently a bug with how we get the versions from the api, i'll send a patch later today

 

[jasonsansone]

Did you restart your services after you upgraded the packages?

In short:

1. restart Monitors on all nodes
2. restart Manager on all nodes
3. restart Meta Data Server
4. restart OSD (step by step, make sure that your ceph status is green again before you restart the next OSD

Yes I did even though it shouldn’t be necessary. I have been up to date on 16.2.6 for months. No new Ceph packages have been released in some time.

 

[dcsapak]

fyi, i sent patches[0] and they are already applied, should be rolling out in a future update..
0: https://lists.proxmox.com/pipermail/pve-devel/2021-November/050833.html

edit: wrong link

 

[jasonsansone]

fyi, i sent patches[0] and they are already applied, should be rolling out in a future update..
0: https://lists.proxmox.com/pipermail/pve-devel/2021-November/050833.html

edit: wrong link

The bug seems to persist in 7.0-14+1.

 

[caramb]

Hello,
I do confirm bug is still in pve-manager 7.0-14+1

At least we know it is a UI bug ; thus purely comestic and not harmful.

 

[caramb]

Hummm... just figured out the oudated warning disppareared on my Promox nodes.
It was there this morning ; it is not anymore.

I performed a package update this morning ; some packages were updated but I did not notice anything relevant to the problem.
I'm currently running version :

pve-manager: 7.0-14+1 (running version: 7.0-14+1/08975a4c)

Perharps one can confirm a fix was released
(I also deep-scrubed my osds because I faced some severe faults on one of the nodes yesterday but it is very unlikely the two are related).

 

[tom]

pve-manager: 7.0-14+1 (running version: 7.0-14+1/08975a4c)

I can confirm the issue, the next version upload will include the fix (in a few days).

 

[caramb]

I can confirm the issue, the next version upload will include the fix (in a few days).

Thank you Tom.

If the fix was not released yet, some magic happened here as the warning is gone.
Running a manual deep-scrub perharps changed something in the API response you base the UI on ?
Anyway, that is not important ; just surprising to me.
(and forgot to mention my homelab uses the pve-nosubscription source)

 

[dan.ger]

I also had this "feature". Remove ceph entirly and reinstall on pve-manager: 7.0-14+1, create new osds and pool. Wait a little bit and the issue outdated osds is resolved.