Ceph OSD Upgrade icon

H

hahosting

Well-Known Member
Aug 20, 2018
48
3
48
Sheffield
www.hahosting.com
Morning folks ive finaly got round to upgrading our hyperconverged cluster from v 5 through v7 and apart from ceph to v quincy which involves me changing from filestore to blue store i have had no issues.

I do however have a yellow icon in the GUI which looks like an ujpgrade icon maybe?

CAn anyone shed any light
 

Attachments

  • Screenshot 2023-01-09 at 10.41.34.png
    Screenshot 2023-01-09 at 10.41.34.png
    2.7 KB · Views: 9
HI Moayad

Yes i have and i have even recreated a row from scratch and the icon still remains.

I was concened it was an error rather than a warning but a ceph -s says everything is fine a ceph versions says they are all the same so im lost.

root@vms-ceph105:~# ceph versions
{
"mon": {
"ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 7
},
"mgr": {
"ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 7
},
"osd": {
"ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 66
},
"mds": {},
"overall": {
"ceph version 16.2.9 (a569859f5e07da0c4c39da81d5fb5675cd95da49) pacific (stable)": 80
}
}
root@vms-ceph105:~#
 
Yes i ran through this section

Restart the OSD Daemon on all Nodes​


Restart all OSDs. Only restart OSDs on one node at a time to avoid loss of data redundancy. To restart all OSDs on a node, run the following command:

systemctl restart ceph-osd.target


Wait after each restart and periodically checking the status of the cluster:

ceph status


It should be in HEALTH_OK or

HEALTH_WARN
noout flag(s) set


Once all OSDs are running with the latest versions, the following warning can appear:

all OSDs are running quincy or later but require_osd_release < quincy


Disallow pre-Quincy OSDs and Enable all new Quincy-only Functionality​

ceph osd require-osd-release quincy
 
ceph -s

root@vms-ceph105:~# ceph -s
cluster:
id: 93cc6f61-f2dd-4346-8813-71de1fa1c221
health: HEALTH_OK

services:
mon: 7 daemons, quorum vms-ceph105,vms-ceph106,vms-ceph110,vms-ceph111,vms-ceph112,vms-ceph113,vms-ceph114 (age 69m)
mgr: vms-ceph106(active, since 38h), standbys: vms-ceph111, vms-ceph110, vms-ceph113, vms-ceph114, vms-ceph112, vms-ceph105
osd: 66 osds: 66 up (since 100m), 66 in (since 27h)

data:
pools: 3 pools, 769 pgs
objects: 5.40M objects, 19 TiB
usage: 52 TiB used, 40 TiB / 92 TiB avail
pgs: 768 active+clean
1 active+clean+scrubbing+deep

io:
client: 32 MiB/s rd, 30 MiB/s wr, 1.03k op/s rd, 1.09k op/s wr

root@vms-ceph105:~#
 
Hello,

Thank you for the output!

Can you also post the output of pveversion -v?

I would restart these services (mon, mgr, osd) to let them run with the latest version.
 
sure

root@vms-ceph105:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-5.15: 7.3-1
pve-kernel-helper: 7.3-1
pve-kernel-5.4: 6.4-20
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.1-1
proxmox-backup-file-restore: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve1
root@vms-ceph105:~#
 
Thank you for the output!

In the above output of pveversion -v you provided, we can see the ceph version still ceph (Pacific). All nodes have the same ceph version output?

Code: 
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1

If you followed our wiki guide `Ceph Pacific to Quincy` [0] should everything ok.

[0] https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
 
hahosting said:
I should also be clear i have not up-graded to quincy yet as i still have some filestore OSD's
Click to expand...
Aha, the cluster still using the Pacific as I understand, if so, why does the previous output show that the MON is using the pacific version? In this case, I would check the PVE source repository. Maybe you have used the `ceph-quincy` repository of the Ceph during the upgrade?
 
Sorry for the misunderstanding, I thought you finished the upgrade to the Ceph quincy, and then you have seen the Yellow icons.

Now, I understand that you're upgrading Ceph Octopus to Pacific [0].

Do restart services (mon, mgr, osd) didn't help?

hahosting said:
I should also be clear i have not up-graded to quincy yet as i still have some filestore OSD's

Could that be the issue?
Click to expand...
Yes, the OSDs should, be of the current Bluestore type.

[0] https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific
 
yes i restarted all services as per the guide, https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific

I have restarted them all again to be safe but no difference.

Alot of these disks are now rebuilt as bluestore and still hav ethe icon which doesnt make sense as they are rebuilt as new as recommeneded in the guide.

I delete the row and recreate using:

ceph-volume lvm create --bluestore --osd-id XXX --block.wal /dev/sdXX --data /dev/sdX
 

Attachments

  • Screenshot 2023-01-09 at 13.58.38.png
    Screenshot 2023-01-09 at 13.58.38.png
    72.6 KB · Views: 9
hahosting said:
Once all OSDs are running with the latest versions, the following warning can appear:

all OSDs are running quincy or later but require_osd_release < quincy


Disallow pre-Quincy OSDs and Enable all new Quincy-only Functionality​

ceph osd require-osd-release quincy
Click to expand...

I do not want to bring unnecessary noise in here with my non-existing experience with Ceph, but one small question:

Is it only a copy-paste "mistake" (from the misunderstandings before) or did you actually execute: ceph osd require-osd-release quincy instead of: ceph osd require-osd-release pacific for your (still) Pacific/16 cluster?

But I also do not know, if this could be the problem at all.
Sorry, if this is indeed unnecessary noise!
 
You must log in or register to reply here.
Top Bottom