[SOLVED] PVE 6.3 upgrade to ceph octopus fails

[nanocosm]

I've a PVE 6.3 Ceph Cluster running 5 nodes. I upgraded the cluster to PVE 6.3 from 6.2 as usual and then followed the octopus upgrade guide Ceph_Nautilus_Octopus

Now after systemctl restart ceph-osd.target the OSDs are marked down



Currently I've no clue how to get out of this. I'm unable to get the OSDs up again. I've attached an osd log starting before restarting ceph-osd.target.

Interestingly I've even tried to re-create an OSD with pveceph osd create /dev/sda. This resulted in a filestore OSD also unusable.

Any idea how to go on further please?

 

[nanocosm]

After reading the log carefully I noticed that line:
2020-12-02T07:52:33.319+0100 7f756541ee00 1 osd.6 14441 check_osdmap_features require_osd_release unknown -> luminous

and then I re-read the Ceph_Luminous_to_Nautilus page where I read that I've had to ceph osd require-osd-release nautilus.

Either I haven't read that while upgrading to Nautilus or that wasn't there previously. However after setting this feature-flag the OSD conversion to Octopus worked, and I was able to start the OSDs.