How to downgrade CEPH from Jewel to Hammer?

[athompso]

I've installed CEPH Jewel, and am having problems with it. I've purged all the CEPH configuration (via "pveceph purge").
Running "pveceph install -version hammer" does not downgrade Jewel to Hammer, although it does alter the apt.sources.d entry. Logically, I then expected to be able to uninstall the CEPH packages, but they all want to trigger removal of pve* as well!
Is there a functional, i.e. working - never mind safe - way to switch back to hammer so I can build a CEPH cluster that actually works?
Thanks,
-Adam

 

[tom]

Downgrade is not really possible, so you have to remove the jewel packages and install hammer.

But Jewel is now tested for Proxmox VE and we will publish a guide in the next days.

 

[athompso]

Since there's a showstopper bug with creating new CEPH clusters in Jewel, I had to figure out how to remove it and reinstall Hammer the hard way:

0. "pveceph install -version hammer" to reset the apt/sources.list.d/ceph.list repository (or just fix it by hand)
1. "dpkg-query --list | grep 10.2.1.5" to find all the matched jewel packages
2. "apt-get purge" all of the packages that can be purged without removing the pve* packages
3. "dpkg --force-depends --purge" all of those packages that have dependency problems
4. "apt-get update" to refresh the pkg cache with hammer packages
5. "apt-get install -f" to fix the dependencies we just broke
6. "pveceph install -version hammer" to install the other required packages
7. a reboot would probably be a really good idea at this point to ensure running processes pick up the correct versions of e..g. PVE/CEPH.pm et al.

 

[dietmar]

Upgrade works for me if you follow the instructions:

https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel

 

[tingjohnson]

Upgrade works for me if you follow the instructions:

https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel

I follow this guide to upgrade my ceph cluster,after I upgrade all nodes,I executed
"ceph osd set require_jewel_osds",then I got this error message.
Invalid command: require_jewel_osds not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd set full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent : set <key>
Error EINVAL: invalid command

Did I miss something in upgrade ?The cluster works fine except this "require_jewel_osds" warning.

 

[spirit]

I follow this guide to upgrade my ceph cluster,after I upgrade all nodes,I executed
"ceph osd set require_jewel_osds",then I got this error message.
Invalid command: require_jewel_osds not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd set full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent : set <key>
Error EINVAL: invalid command

Did I miss something in upgrade ?The cluster works fine except this "require_jewel_osds" warning.

are you sure that all package have been upgraded ? including "ceph" command ?

dpkg -l|grep ceph ?

 

[tingjohnson]

are you sure that all package have been upgraded ? including "ceph" command ?

dpkg -l|grep ceph ?

Here is my ceph version info
root@pvnews3:~# dpkg -l|grep ceph
ii ceph-base 10.2.5-1~bpo80+1 amd64 common ceph daemon libraries and management tools
ii ceph-common 10.2.5-1~bpo80+1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fs-common 10.2.5-1~bpo80+1 amd64 common utilities to mount and interact with a ceph file system
ii ceph-fuse 10.2.5-1~bpo80+1 amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 10.2.5-1~bpo80+1 amd64 metadata server for the ceph distributed file system
ii ceph-mon 10.2.5-1~bpo80+1 amd64 monitor server for the ceph storage system
ii ceph-osd 10.2.5-1~bpo80+1 amd64 OSD server for the ceph storage system
ii libcephfs1 10.2.5-1~bpo80+1 amd64 Ceph distributed file system client library
ii python-ceph 10.2.5-1~bpo80+1 amd64 Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.5-1~bpo80+1 amd64 Python libraries for the Ceph libcephfs library
root@pvnews3:~# ceph -v
ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
root@pvnews3:~# ceph osd set require_jewel_osds
Invalid command: require_jewel_osds not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd set full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent : set <key>
Error EINVAL: invalid command

 

[athompso]

Upgrade works for me if you follow the instructions:

https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel

Dietmar, as I've said elsewhere - this wasn't an upgrade. I expect the upgrade to work fine. What's broken for me is a brand-new, fresh install of Jewel on a fresh install of 4.3 immediately updated to 4.4.

 

[tingjohnson]

After I reboot all my ceph nodes,The issue is gone.I think you need to reboot all your ceph nodes after upgrade.
Since Jewel support nbd,I am wondering if proxmox would support nbd,It's more functions,less performance,better stability(perhaps).