Ceph Squid upgrade to 19.2.1

[deda_lebeda]

Based on the 8.4 release notes and https://tracker.ceph.com/issues/70390 there is an issue with OSDs (currently the issue is affecting also older deployments).

There is an advice to update ASAP, but when to expect the OSDs to fail? Before update, during the update procedure when some nodes are at higher version or with old OSDs at full 19.2.1 cluster?

What is recommended upgrade procedure? Update one node, destroy its OSDs and recreate them and continue with other nodes, or update all nodes to same Proxmox and Ceph version and then start OSD recreate process?

 

[EllerholdAG]

Upgrade Proxmox and ceph asap.
Destroy one OSD at a time and recreate it. Wait for ceph to finish rebalancing before destroying the next OSD.
You ONLY have to do the last part for OSDs created when you had 19.2.0 installed. OSDs created earlier are fine.

 

[itNGO]

This is only an issue if using Erasure Coding, isn't it?

 

[deda_lebeda]

Based on this comment https://tracker.ceph.com/issues/70390#note-20 there are also affected deployments prior Squid.

 

[deda_lebeda]

This is only an issue if using Erasure Coding, isn't it?

If there is EC pool it will bring down the whole OSD, so my radosgw EC can cause headaches for wholel cluster

 

[deda_lebeda]

Upgrade Proxmox and ceph asap.

Just there is another issue with I/O slow operations at new version of Ceph. https://forum.proxmox.com/threads/c...eriencing-slow-operations-in-bluestore.164856

 

[EllerholdAG]

This is just new reporting, no new bug etc was introduced.

 

[SteveITS]

This is only an issue if using Erasure Coding, isn't it?

Per https://pve.proxmox.com/wiki/Roadmap#8.4-known-issues they say "seems to affect EC pools in particular" which implies it can affect non-EC.

Simply upgrading doesn't fix anything. The workaround/patch is to turn off that buggy setting by default, but the OSD needs to be created with the setting off, otherwise it is still on.