Ceph 18.2 Reef Available and Ceph 16.2 Pacific soon to be EOL

[t.lamprecht]

Hi Community!

The recently released Ceph 18.2 Reef is now available on all Proxmox Ceph repositories to install or upgrade.

Upgrades from Quincy to Reef:
You can find the upgrade how to here: https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

New Installation of Reef:
Use the updated ceph installation wizard available with the recently released pve-manager version 8.0.6 (available on the pvetest repository at time of writing).

Current State:
We ran and tested the release since a few weeks internally and found no major issue.
Ceph Quincy will stay supported until mid-2024 for the time being.
Known minor issues are the same as for Ceph Quincy: there are still some issues with ancient CPUs without SSE 4.1 support (fixed), and the ceph-dashboard, (which Proxmox VE doesn't rely on) seems to have some python compatibility issues on Debian Bookworm based releases.

Road to Enterprise Stability: Edit: Since 2023-11-23 Ceph Reef is deemed stable and available on the Enterprise repository.
Our further plan is to lift the preview state and provided Reef as fully supported Ceph version once we got even more test time and feedback from QA, and naturally we would be happy to hear about the observations from our great community! Once we deem the Ceph 18.2 Reef release, and it's integration into Proxmox VE fully production ready, we'll also populate the Ceph Reef enterprise repository.

FYI: We also plan to publish a new Ceph Benchmark paper still this year, comparing among other things the performance of Ceph Quincy and Reef on state-of-the-art hardware. Let's see if the initial increase in IOPS and bandwidth performance we observed on some test systems after upgrading to Reef is confirmed there as well.

Reminder: Old Ceph 16.2 Pacific Going EOL Soon:
Please also remember that Ceph 16.2 Pacific is going to be end of life (EOL) after 2023-10 (next month), so you should upgrade any existing Ceph Pacific setups to Ceph Quincy rather sooner than later. Note that this is also a required prerequisite for upgrading from Proxmox VE 7 to Proxmox VE 8, which only supports hosting Quincy and newer releases as Ceph-Server. See the respective upgrade how-to: https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy.

EDIT: Changed the title to reflect that this release is now fully stable.

 

[jsterr]

Cant wait to try out, thanks for the great work!

 

[ness1602]

I've updated one 3-node cluster, and so far the procedure has been a breeze.

 

[scyto]

Well that's sweet UI right there... it differentiates the nodes where i have run apt update from the ones where i also did apt upgrade! nice.



this ones nice too

 

[scyto]

yes i hit the documented "If the managers did not automatically restart with the monitors" (restartin them as documented worked)

 

[scyto]

I was very confused by all the talk of ranks in the MDS section...

What i think you wanted was:

1. make sure there is only one active daemon per filesystem, if there is more issue the set command
2. once you only have one per filesystem stop all MDS listed as up:standby
3. when they are stopped restart the up:active
4. then restart the rest one by one

which is what i did and it all worked great

In the UI, the command line and even using ceph -s i never once saw the word rank anywhere - confusing for new person (i still don't know what a rank is)

 

[t.lamprecht]

Many thanks for your feedback!

I was very confused by all the talk of ranks in the MDS section...

FWIW, it quite early links to the ceph terminology page that describes what is meant with "rank" in the CephFS context:
https://docs.ceph.com/en/latest/cephfs/standby/#terminology
As the ceph docs make extensive use of that term, we also use it in the upgrade guide.

But, your feedback has been heard, and it might be indeed that users that are new to ceph, and get exposed to it mostly through Proxmox VE, might be confused by that term and overlook the (a bit lengthy) terminology explanation.

make sure there is only one active daemon per filesystem, if there is more issue the set command

Almost. A single CephFS can have more ranks, i.e., more than on MDS handling parts of its operation (mostly used for bigger instances with lots of usage of such a CephFS). So yes, by default it means that. But if an admin had to scale a CephFS up, e.g. due to heavy demand or just because it's a huge instance with like hundreds of TB usable space.

So, I think it's important to keep using the term "rank" here, but I tried to reword that section a bit to hopefully make it more clear what is meant here.

Maybe we can look into adding the "rank" term on the UI too, ideally such that it also helps on checking the status during when one follows the upgrade how-to.

 

[jsterr]

Well that's sweet UI right there... it differentiates the nodes where i have run apt update from the ones where i also did apt upgrade! nice.

View attachment 55675

this ones nice too

View attachment 55677

i hope you never used apt upgradeon your proxmox nodes!

Edit: Our upgrade to reef worked without any problems, no downtime, no hassle

 

[scyto]

FWIW, it quite early links to the ceph terminology page that describes what is meant with "rank" in the CephFS context:

Thanks for explaining and that link - maybe you Europeans read all the docs before you begin... not here in the US (it was a shock when i moved here).

Maybe in the wiki you created, which is all I read, add in the assumptions that link? 'aka assume you are familiar with ceph terms <link>` - because others will do what i did, because you have created such darn good UI that one can set all of this up with no need to read the docs

Maybe we can look into adding the "rank" term on the UI too

thats a great idea, actually where i really got hung up was the statements about rank relative to ceph -s - which never mentioned them, this would seem to be flaw in the ceph project not using their own terminology...

 

[scyto]

i hope you never used apt upgrade on your proxmox nodes!

yup on all of them! as that is what the UI does.... (hint my nodes were fully upgraded before i did the ceph upgrade anyway so i knew exactly what was and wasn't going to be upgraded - it was only the items from the ceph deb list)

 

[fabian]

getting a bit OT, but: the UI will use "pveupgrade", which in turn runs "apt-get dist-upgrade", not "apt upgrade"

 

[scyto]

Thanks that sent me down this rabbit hole and I learnt something important about apt vs apt-get https://askubuntu.com/questions/770135/apt-full-upgrade-versus-apt-get-dist-upgrade

 

[Zerstoiber]

Derailing the Ceph-thread ;-<

The difference is not "apt" vs "apt-get", it's the difference between the very much recommended "apt dist-upgrade" vs. the simpler "apt upgrade"
I might have to save this for quoting again and again ;-)

TLDR: forget "apt upgrade", use "apt dist-upgrade"!


dist-upgrade and full-upgrade are the same thing, evenmoreso the same in "apt" or "apt-get" as they reference the same included function.

https://sources.debian.org/src/apt/2.7.6/cmdline/apt.cc/

{"upgrade", &DoUpgrade, _("upgrade the system by installing/upgrading packages")},
{"full-upgrade", &DoDistUpgrade, _("upgrade the system by removing/installing/upgrading packages")},
{"dist-upgrade", &DoDistUpgrade, nullptr},

https://sources.debian.org/src/apt/2.7.6/cmdline/apt-get.cc/

{"upgrade", &DoUpgrade, _("Perform an upgrade")},
{"dist-upgrade", &DoDistUpgrade, _("Distribution upgrade, see apt-get(8)")},
{"full-upgrade", &DoDistUpgrade, nullptr},

 

[BenediktS]

We updated the smaller of our two ceph clusters.

Update went smooth.
And it's running fine for 2 weeks now.

 

[Ivanmarrese]

So, the sentence below (taken from Official Ceph doc site ) should not be taken into consideration... Right?


Important

We are unable to build Ceph on Debian stable (bookworm) for the 18.2.0release because of Debian bughttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129. We will build assoon as this bug is resolved in Debian stable.

last updated 2023 Aug 04

 

[t.lamprecht]

No, you can ignore that as long as you use our repos. Ceph upstream being affected here is IIRC mostly an issue in their CI/builders.

 

[Ivanmarrese]

Thank you Thomas! I'm going to upgrade soon

 

[Firebat]

Upgrade went smooth on a 3-node cluster, lets see if it stays that way. No need for manual restart of the managers here.

 

[Csaba Tóth]

When will Ceph 18.2.1 be added to the proxmox repo? I want to wait until the first patch is released before upgrading from Ceph 17 to 18, but it was released weeks ago.

 

[tom]

When will Ceph 18.2.1 be added to the proxmox repo?

Already available in test repo.

http://download.proxmox.com/debian/ceph-reef/dists/bookworm/test/binary-amd64/