Hyperconverged pve6to7 Ceph multiple running versions detected

[Russell Ault]

I'm finally getting around to upgrading to PVE 7 on my three-node hyperconverged cluster, and I'm following the upgrade guide in the Wiki. I installed all the PVE 6.4 upgrades on all three nodes, then upgraded the cluster to Ceph Octopus. Once that was done, I ran pve6to7 on one of the three nodes, and when it came back clean (well, one warning because I was just shy of 4GB of free disk space, but I was sure I'd be close enough) I upgraded the node to PVE7. This went totally smoothly (as smoothly as any of the other buster-bullseye upgrades I've done recently) and the node and its VMs are running happily.

Great! Time for the second node. I run pve6to7 and, in addition to the free space warning, I'm now getting four other warnings as well:

= CHECKING HYPER-CONVERGED CEPH STATUS =

INFO: hyper-converged ceph setup detected!
INFO: getting Ceph status/health information..
PASS: Ceph health reported as 'HEALTH_WARN' with a single failing check and 'noout' flag set.
INFO: getting Ceph daemon versions..
WARN: multiple running versions detected for daemon type monitor!
WARN: multiple running versions detected for daemon type manager!
SKIP: no running instances detected for daemon type MDS.
WARN: multiple running versions detected for daemon type OSD!
WARN: overall version mismatch detected, check 'ceph versions' output for details!
PASS: 'noout' flag set to prevent rebalancing during cluster-wide upgrades.
INFO: checking Ceph config..

For what it's worth, the Datacenter->Ceph panel is showing green checkmarks for all three monitors and managers, and ceph versions says this:

{
    "mon": {
        "ceph version 15.2.15 (4b7a17f73998a0b4d9bd233cda1db482107e5908) octopus (stable)": 2,
        "ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)": 1
    },
    "mgr": {
        "ceph version 15.2.15 (4b7a17f73998a0b4d9bd233cda1db482107e5908) octopus (stable)": 2,
        "ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)": 1
    },
    "osd": {
        "ceph version 15.2.15 (4b7a17f73998a0b4d9bd233cda1db482107e5908) octopus (stable)": 4,
        "ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)": 2
    },
    "mds": {},
    "overall": {
        "ceph version 15.2.15 (4b7a17f73998a0b4d9bd233cda1db482107e5908) octopus (stable)": 8,
        "ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)": 4
    }
}

Everything seems happy otherwise, but I'm concerned about proceeding without understanding what these warnings are about. Any insights?

Thanks!

 

[Russell Ault]

Okay, so, digging a little deeper, it appears that the Proxmox Ceph 15.2.15 packages for Buster and Bullseye are just different enough (they have different CEPH_GIT_VERs) to trigger this warning after the first node has been upgraded. I'm not sure if this constitutes a bug or not, since the information is technically correct, but the warning itself is effectively spurious (because the "multiple running versions" it's warning about are an unavoidable part of the cluster upgrade process).