Upgrading pve 7.2 -> 7.4 and ceph pacific -> quincy failed

[ronzle]

Hi,

i have a cluster of 3 compute nodes and 3 storage nodes.
I wanted to upgrade to pve 7.4 and ceph quincy.
Followed the official documentation https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
All went ok until i restarted the osds on one of the storage nodes.
some are upgraded and some not (picture) ...
Seems to me the monitors prevents the upgrading ?
Can anyone help with this ?
root@storage01:~# ceph -s
cluster:
id: 2728e9f8-7dc0-4da3-bf91-f7bf63c5b428
health: HEALTH_WARN
noout flag(s) set
19 osds down
1 host (19 osds) down
all OSDs are running pacific or later but require_osd_release < pacific
Degraded data redundancy: 1147095/3510840 objects degraded (32.673%), 2071 pgs degraded, 2071 pgs undersized
34 daemons have recently crashed

services:
mon: 6 daemons, quorum compute01,storage01,storage02,compute02,compute03,storage03 (age 18m)
mgr: compute01(active, since 11h), standbys: compute02
osd: 57 osds: 38 up (since 13h), 57 in (since 11h); 42 remapped pgs
flags noout

data:
pools: 4 pools, 2113 pgs
objects: 1.17M objects, 4.4 TiB
usage: 14 TiB used, 48 TiB / 62 TiB avail
pgs: 1147095/3510840 objects degraded (32.673%)
23185/3510840 objects misplaced (0.660%)
2071 active+undersized+degraded
42 active+clean+remapped

io:
client: 194 KiB/s rd, 1.6 MiB/s wr, 13 op/s rd, 213 op/s wr


config

root@storage01:~# cat /etc/ceph/ceph.conf
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.10.10.10/24
fsid = 2728e9f8-7dc0-4da3-bf91-f7bf63c5b428
mon_allow_pool_delete = true
mon_host = 10.10.10.10 10.10.10.20 10.10.10.21 10.10.10.11 10.10.10.12 10.10.10.22
ms_bind_ipv4 = true
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.10.10.10/24

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.compute01]
public_addr = 10.10.10.10

[mon.compute02]
public_addr = 10.10.10.11

[mon.compute03]
public_addr = 10.10.10.12

[mon.storage01]
public_addr = 10.10.10.20

[mon.storage02]
public_addr = 10.10.10.21

[mon.storage03]
public_addr = 10.10.10.22


I'm running out of ideas

 

[Alwin Antreich]

I suppose you fixed the issue? Would you share some information for posterity?

 

[Maximiliano]

Hello,

Did you see the note at [1]? It mentions specifically your issue.

[1] https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy#Restart_the_OSD_Daemon_on_all_Nodes