Ceph 19.2 Squid Available as Technology Preview and Ceph 17.2 Quincy soon to be EOL

[t.lamprecht]

Hi Community!

The recently released Ceph 19.2 Squid is now available on the Proxmox Ceph test and no-subscription repositories to install or upgrade.

Upgrades from Reef to Squid:
You can find the upgrade how-to here: https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid

New Installation of Reef:
Use the updated ceph installation wizard available with the recently released pve-manager version 8.2.8 (available on the pvetest repository at time of writing).

Current State:
We ran and tested the release since a few weeks internally and found no major issue.
Ceph 18.2 Reef will stay supported until mid-2025 for the time being.

Road to Enterprise Stability:
Our further plan is to lift the preview state and provided Squid as fully supported Ceph release once we got even more test time and feedback from QA, and naturally we would be happy to hear about the observations from our great community! Once we deem the Ceph 19.2 Squid release, and it's integration into Proxmox VE fully production ready, we'll also populate the Ceph Squid enterprise repository.

Reminder: Old Ceph 17.2 Quincy Going EOL Soon:
Please also remember that Ceph 17.2 Pacific is going to be end of life (EOL) soon and received no update since a while already. So you should upgrade any existing Ceph Quincy setups to Ceph Reef, or soon also Ceph Squid, rather sooner than later. See the upgrade how-to: https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef.

We welcome your feedback!

 

[ness1602]

I've upgraded one 4-node cluster,no problems for now.

 

[Nemesiz]

Squid have problems with orchestrator and dashboard functionality. As one created issue https://tracker.ceph.com/issues/68657
In my test lab I had the same thing. I`m just saying for just in case.

 

[t.lamprecht]

Squid have problems with orchestrator and dashboard functionality. As one created issue https://tracker.ceph.com/issues/68657
In my test lab I had the same thing. I`m just saying for just in case.

For completeness' sake: Proxmox VE's Ceph management stack does not depend on any of those two, but some might use them nonetheless, so thanks for posting that hint as heads up for those users.

 

[itNGO]

Just to be sure... the pure upgrade would not enable any "Crimson" feature for now? It's just the latest version and for anything else manual configuration will be necessary?

 

[t.lamprecht]

Just to be sure... the pure upgrade would not enable any "Crimson" feature for now? It's just the latest version and for anything else manual configuration will be necessary?

No, Crimosn is a technology preview in the Ceph 19.2 Squid release and is currently not enabled on build in our packaging.
We will take a closer look at how to expose it as opt-in co-installable with ceph-osd in the future, at the moment we're more interested in ensuring Ceph 19.2 Squid gets as stable as is.

 

[yangsm]

I hope to build a separate version of Crimosn to test its performance.

 

[jsterr]

We updated one of our clusters already, no issues so far (no downtime, everything correct).

 

[Neobin]

This is an up-to-date (no subscription) PVE installation with PBS alongside, but without a Ceph full-installation; only the default ootb parts:

Spoiler: apt list --upgradable

Listing... Done
ceph-common/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
ceph-fuse/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
libcephfs2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librados2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
libradosstriper1/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librbd1/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librgw2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-ceph-argparse/stable 19.2.0-pve1 all [upgradable from: 18.2.4-pve3]
python3-ceph-common/stable 19.2.0-pve1 all [upgradable from: 18.2.4-pve3]
python3-cephfs/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rados/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rbd/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rgw/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]

Spoiler: apt full-upgrade

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  autoconf automake autotools-dev cpp cpp-12 gcc gcc-12 libabsl20220623 libaom3 libasan8 libatomic1 libavif15 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdav1d6 libde265-0 libdeflate0 libgav1-1 libgcc-12-dev libgd3
  libgomp1 libheif1 libisl23 libitm1 libjbig0 liblerc4 liblsan0 libltdl-dev liblua5.3-dev libmpc3 libmpfr6 libncurses-dev libnsl-dev libpkgconf3 libquadmath0 librav1e0 libreadline-dev libsvtav1enc1 libtiff6 libtirpc-dev libtool
  libtool-bin libtsan2 libubsan1 libwebp7 libx265-199 libxpm4 libyuv0 linux-libc-dev lua-any lua-sec lua-socket lua5.1 luarocks m4 manpages-dev pkg-config pkgconf pkgconf-bin rpcsvc-proto unzip zip
The following packages will be upgraded:
  ceph-common ceph-fuse libcephfs2 librados2 libradosstriper1 librbd1 librgw2 python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw
13 upgraded, 65 newly installed, 0 to remove and 0 not upgraded.
Need to get 106 MB of archives.
After this operation, 254 MB of additional disk space will be used.

Is it expected, that there will get 65 new packages get pulled in?

 

[fiona]

Hi,

This is an up-to-date (no subscription) PVE installation with PBS alongside, but without a Ceph full-installation; only the default ootb parts:

Spoiler: apt list --upgradable

Listing... Done
ceph-common/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
ceph-fuse/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
libcephfs2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librados2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
libradosstriper1/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librbd1/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
librgw2/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-ceph-argparse/stable 19.2.0-pve1 all [upgradable from: 18.2.4-pve3]
python3-ceph-common/stable 19.2.0-pve1 all [upgradable from: 18.2.4-pve3]
python3-cephfs/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rados/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rbd/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]
python3-rgw/stable 19.2.0-pve1 amd64 [upgradable from: 18.2.4-pve3]

Spoiler: apt full-upgrade

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  autoconf automake autotools-dev cpp cpp-12 gcc gcc-12 libabsl20220623 libaom3 libasan8 libatomic1 libavif15 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdav1d6 libde265-0 libdeflate0 libgav1-1 libgcc-12-dev libgd3
  libgomp1 libheif1 libisl23 libitm1 libjbig0 liblerc4 liblsan0 libltdl-dev liblua5.3-dev libmpc3 libmpfr6 libncurses-dev libnsl-dev libpkgconf3 libquadmath0 librav1e0 libreadline-dev libsvtav1enc1 libtiff6 libtirpc-dev libtool
  libtool-bin libtsan2 libubsan1 libwebp7 libx265-199 libxpm4 libyuv0 linux-libc-dev lua-any lua-sec lua-socket lua5.1 luarocks m4 manpages-dev pkg-config pkgconf pkgconf-bin rpcsvc-proto unzip zip
The following packages will be upgraded:
  ceph-common ceph-fuse libcephfs2 librados2 libradosstriper1 librbd1 librgw2 python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw
13 upgraded, 65 newly installed, 0 to remove and 0 not upgraded.
Need to get 106 MB of archives.
After this operation, 254 MB of additional disk space will be used.

Is it expected, that there will get 65 new packages get pulled in?

thanks for the report! What if you use --no-install-recommends?

From a quick glance, it seems like the "culprit" is librgw2 -Depends-> liblua5.3-dev -Recommends-> libtool-bin

root@pve8a1 ~ # aptitude why autoconf
i   ceph-common   Depends    python3-rgw (= 19.2.0-pve1)
i A python3-rgw   Depends    librgw2 (>= 19.2.0-pve1)  
i A librgw2       Depends    liblua5.3-dev              
i A liblua5.3-dev Recommends libtool-bin                
i A libtool-bin   Depends    libtool (>= 2.4.7-7~deb12u1)
i A libtool       Recommends libltdl-dev                
i A libltdl-dev   Depends    automake-1.16              
i A automake      Provides   automake-1.16              
i A automake      Depends    autoconf

 

[jsterr]

This one failed for me ( command from: https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid)

root@training2:~# ceph telemetry on
Error EPERM: Telemetry data is licensed under the Community Data License Agreement - Sharing - Version 1.0 (https://cdla.io/sharing-1-0/).
To enable, add '--license sharing-1-0' to the 'ceph telemetry on' command.

It worked with:

root@training2:~# ceph telemetry on --license sharing-1-0
Telemetry is on.
Some channels are disabled, please enable with:
`ceph telemetry enable channel perf`

 

[aaron]

This one failed for me

Thanks, I fixed the docs!

 

[jsterr]

I also noticed that if you do a fresh install, you cant find the repository in the web-ui.
It this supposed to be so?

root@pve1:~# pveceph install --repository no-subscription --version
quincy  reef

 

[aaron]

I also noticed that if you do a fresh install, you cant find the repository in the web-ui.

From the first post:

New Installation of Reef:
Use the updated ceph installation wizard available with the recently released pve-manager version 8.2.8 (available on the pvetest repository at time of writing).

Needs new pve-manager 8.2.8 which is in pvetest only for now.

 

[Neobin]

Hi,

thanks for the report! What if you use --no-install-recommends?

From a quick glance, it seems like the "culprit" is librgw2 -Depends-> liblua5.3-dev -Recommends-> libtool-bin

root@pve8a1 ~ # aptitude why autoconf
i   ceph-common   Depends    python3-rgw (= 19.2.0-pve1)
i A python3-rgw   Depends    librgw2 (>= 19.2.0-pve1) 
i A librgw2       Depends    liblua5.3-dev             
i A liblua5.3-dev Recommends libtool-bin               
i A libtool-bin   Depends    libtool (>= 2.4.7-7~deb12u1)
i A libtool       Recommends libltdl-dev               
i A libltdl-dev   Depends    automake-1.16             
i A automake      Provides   automake-1.16             
i A automake      Depends    autoconf

Thank you. This gets the count down quite a bit:

Spoiler: apt full-upgrade --no-install-recommends

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libc-dev-bin libc6-dev libcrypt-dev liblua5.3-dev libncurses-dev libnsl-dev libreadline-dev libtirpc-dev linux-libc-dev lua-any lua5.1 luarocks rpcsvc-proto unzip zip
The following packages will be upgraded:
  ceph-common ceph-fuse libcephfs2 librados2 libradosstriper1 librbd1 librgw2 python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw
13 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
Need to get 52.6 MB of archives.
After this operation, 53.9 MB of additional disk space will be used.

As you stated, the dependencies from librgw2 of liblua5.3-dev and luarocks(?) seem to be it:

Spoiler: apt-rdepends librgw2 -p --state-show=NotInstalled --state-follow=NotInstalled

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
librgw2
  Depends: liblua5.3-dev [NotInstalled]
  Depends: luarocks [NotInstalled]
liblua5.3-dev
  Depends: libc-dev [NotInstalled]
  Depends: libc6-dev [NotInstalled]
  Depends: libreadline-dev [NotInstalled]
libc-dev
libc6-dev
  Depends: libc-dev-bin (= 2.36-9+deb12u8) [NotInstalled]
  Depends: libcrypt-dev [NotInstalled]
  Depends: libnsl-dev [NotInstalled]
  Depends: linux-libc-dev [NotInstalled]
  Depends: rpcsvc-proto [NotInstalled]
libc-dev-bin
libcrypt-dev
libnsl-dev
  Depends: libtirpc-dev [NotInstalled]
libtirpc-dev
linux-libc-dev
rpcsvc-proto
libreadline-dev
  Depends: install-info [NotInstalled]
  Depends: libncurses-dev [NotInstalled]
install-info
libncurses-dev
  Depends: libc-dev [NotInstalled]
  Depends: libc6-dev [NotInstalled]
luarocks
  Depends: liblua5.1-dev [NotInstalled]
  Depends: liblua5.2-dev [NotInstalled]
  Depends: liblua5.3-dev [NotInstalled]
  Depends: lua-any [NotInstalled]
  Depends: lua5.1 [NotInstalled]
  Depends: lua5.2 [NotInstalled]
  Depends: lua5.3 [NotInstalled]
  Depends: unzip [NotInstalled]
  Depends: zip [NotInstalled]
liblua5.1-dev
liblua5.2-dev
  Depends: libc-dev [NotInstalled]
  Depends: libc6-dev [NotInstalled]
  Depends: liblua5.2-0 (= 5.2.4-3) [NotInstalled]
  Depends: libreadline-dev [NotInstalled]
liblua5.2-0
lua-any
  Depends: lua [NotInstalled]
  Depends: lua5.1 [NotInstalled]
lua
lua5.1
lua5.2
lua5.3
unzip
zip

 

[t.lamprecht]

Is it expected, that there will get 65 new packages get pulled in?

As you stated, the dependencies from librgw2 of liblua5.3-dev and luarocks(?) seem to be it:

This should be fixed now with version 19.2.0-pve2, which we just uploaded to the test/no-subscription ceph-squid repo.
Many thanks again for bringing this to our attention, I also created a pull request upstream so that the broader Ceph ecosystem can profit from your findings.

 

[Neobin]

This should be fixed now with version 19.2.0-pve2, which we just uploaded to the test/no-subscription ceph-squid repo.
Many thanks again for bringing this to our attention, I also created a pull request upstream so that the broader Ceph ecosystem can profit from your findings.

Thank you. Currently I am seeing the new versions only in the test repository. Not at all that I am in a rush, but wanted to mention it, since it seems as not intended?

 

[t.lamprecht]

Not at all that I am in a rush, but wanted to mention it, since it seems as not intended?

It was imported to no-subscription but not yet synced to the CDN, I triggered the sync now.

 

[Neobin]

This should be fixed now with version 19.2.0-pve2, which we just uploaded to the test/no-subscription ceph-squid repo.
Many thanks again for bringing this to our attention, I also created a pull request upstream so that the broader Ceph ecosystem can profit from your findings.

It was imported to no-subscription but not yet synced to the CDN, I triggered the sync now.

Thank you very much.

Now it is all as I would expect it:

The following packages will be upgraded:
  ceph-common ceph-fuse libcephfs2 librados2 libradosstriper1 librbd1 librgw2 python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw
13 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

 

[Neobin]

Tiny oversight in the label (18 -> 19):

http://download.proxmox.com/debian/ceph-squid bookworm/no-subscription amd64 Packages
release o=Proxmox,a=stable,n=bookworm,l=Proxmox Ceph 18 Squid Debian Repository,c=no-subscription,b=amd64
origin download.proxmox.com