Ceph 18.2 Reef Available and Ceph 16.2 Pacific soon to be EOL

[tuxis]

Please note that after an upgrade of a cluster from Octopus to Reef (18.2.0), which has been running since PVE 5.4, we've seen issues with this Ceph version. We seem to be hitting bugs where the OSD is unable to (quickly) find aligned 64k blocks for rocksdb. This has been fixed in 18.2.1 (by removing the requirement for the 64k block to be aligned: https://github.com/ceph/ceph/pull/54772).

If you are suffering from this, it helps if you add more OSD's so that there is more free space available. It also helps to offline compact your kv-store. We see quite a few 'laggy' pgs since the upgrade. The first night (before adding disks and compacting the kv-store) was horrible, with OSD's being flagged down because they were too slow.

You might notice the following log messages:

bluestore(/var/lib/ceph/osd/ceph-3) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.574412823s, txc = 0x55ddf99bd500

We've done offline compactation by running:

ceph osd set noout
systemctl stop ceph-osd@$id
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-$id/ compact
systemctl start ceph-osd@$id
ceph osd unset noout

Please keep your failure domain in mind before taking OSD's offline!

 

[tuxis]

Just upgraded to 18.2.1 which is in Enterprise now.

 

[Amol.pande]

Hi Community!

The recently released Ceph 18.2 Reef is now available on all Proxmox Ceph repositories to install or upgrade.

Upgrades from Quincy to Reef:
You can find the upgrade how to here: https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

New Installation of Reef:
Use the updated ceph installation wizard available with the recently released pve-manager version 8.0.6 (available on the pvetest repository at time of writing).

Current State:
We ran and tested the release since a few weeks internally and found no major issue.
Ceph Quincy will stay supported until mid-2024) for the time being.
Known minor issues are the same as for Ceph Quincy: there are still some issues with ancient CPUs without SSE 4.1 support (fixed), and the ceph-dashboard, (which Proxmox VE doesn't rely on) seems to have some python compatibility issues on Debian Bookworm based releases.

Road to Enterprise Stability: Edit: Since 2023-11-23 Ceph Reef is deemed stable and available on the Enterprise repository.
Our further plan is to lift the preview state and provided Reef as fully supported Ceph version once we got even more test time and feedback from QA, and naturally we would be happy to hear about the observations from our great community! Once we deem the Ceph 18.2 Reef release, and it's integration into Proxmox VE fully production ready, we'll also populate the Ceph Reef enterprise repository.

FYI: We also plan to publish a new Ceph Benchmark paper still this year, comparing among other things the performance of Ceph Quincy and Reef on state-of-the-art hardware. Let's see if the initial increase in IOPS and bandwidth performance we observed on some test systems after upgrading to Reef is confirmed there as well.

Reminder: Old Ceph 16.2 Pacific Going EOL Soon:
Please also remember that Ceph 16.2 Pacific is going to be end of life (EOL) after 2023-10 (next month), so you should upgrade any existing Ceph Pacific setups to Ceph Quincy rather sooner than later. Note that this is also a required prerequisite for upgrading from Proxmox VE 7 to Proxmox VE 8, which only supports hosting Quincy and newer releases as Ceph-Server. See the respective upgrade how-to: https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy.

EDIT: Changed the title to reflext that the

Dear t.lamprecht ,

Good day,

I am facing an issue can you help me?

First of all, I am sorry to ask you here.

I am planning this Proxmox in my environment. But before that I want to do a demo. If it is done well. So then we are going to implement this also, with your subscription. So I have to install ceph. But that is not happening. I am keeping everything in snap attachment, will you help me with this issue please?

Thanks in Advanced
Amol

 

[apollo13]

18.2.2 (https://github.com/proxmox/ceph/commits/master/) has some nice keyring fixes. Any ideas when they will make it into the enterprise repos?

 

[Falk R.]

Dear t.lamprecht ,

Good day,

I am facing an issue can you help me?

First of all, I am sorry to ask you here.

I am planning this Proxmox in my environment. But before that I want to do a demo. If it is done well. So then we are going to implement this also, with your subscription. So I have to install ceph. But that is not happening. I am keeping everything in snap attachment, will you help me with this issue please?

Thanks in Advanced
Amol

You can buy a Subscription or deactivate the Enterprise Repos and add No-Subscription Repo.

 

[t.lamprecht]

18.2.2 (https://github.com/proxmox/ceph/commits/master/) has some nice keyring fixes. Any ideas when they will make it into the enterprise repos?

We slowed down moving that version along after we saw an odd report about OSD not coming up in this post: https://forum.proxmox.com/threads/after-updating-ceph-18-2-2-each-osds-never-start.144621/

But I did not get any further feedback yet there, and we could not reproduce this in our test and production clusters at all yet.. So, if no new info about this potential issue comes up over the next days, it's rather safe to call it a fluke, and in that case we plan to move 18.2.2 to the enterprise repository until the middle of next week.