Proxmox as CEPH Nautilus client
- Thread starter janos
- Start date
This depends on the 'ceph osd set-require-min-compat-client' setting and defines the oldest client that can connect. Here you have to keep the kernel client in mind too, so luminous or jewel if stock debian ceph packages are used (fresh install).
Currently we have mimic on CentOS, min client is jewel. So, theoritically ceph client in proxmox can connect to nautilus ceph, if it allow it.
thanks.
thanks.
https://pve.proxmox.com/wiki/Package_Repositories
This is Proxmox VE’s main Ceph repository and holds the Ceph packages for production use. You can also use this repository to update only the Ceph client.
File /etc/apt/sources.list.d/ceph.list
deb http://download.proxmox.com/debian/ceph-nautilus buster main
I have no issues getting to connect to Nautilus. It appears that Proxmox is still using the Jewel client which limits the ability to use some settings.
It would appear that 6.0-4 is still using the jewel client.
Error EPERM: cannot set require_min_compat_client to luminous: 3 connected client(s) look like jewel (missing 0x800000000000000);
A fresh Proxmox VE 6 or buster installation uses the luminous client.
Code:
:~# dpkg -l | grep ceph
ii ceph-common 12.2.11+dfsg1-2.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 12.2.11+dfsg1-2.1 amd64 FUSE-based client for the Ceph distributed file system
ii libcephfs2:amd64 12.2.11+dfsg1-2.1 amd64 Ceph distributed file system client library
ii python-cephfs 12.2.11+dfsg1-2.1 amd64 Python libraries for the Ceph libcephfs libraryOdd I show the file versions as 14.2.1-pve2 for these files.
I installed 5.4 Fresh with no Ceph upgraded to 6.0-1 Beta install Ceph and then upgraded to 6.0-4 release
Once you installed ceph explicitly, then it is 14.2.1. The packages I mentioned are from a stock install, without any extra package installation.
There is still something amiss. I have upgraded my cluster to
The ceph-packages are all nautilus:
Now, in order to use the balancer for ceph (https://docs.ceph.com/docs/mimic/mgr/balancer), I need to set the minimum client allowed for my cluster to luminous:
What am I missing? The only clients for this cluster are the VMs using it for their disks.
Bash:
~# pveversion
pve-manager/6.0-9/508dcee0 (running kernel: 5.0.21-3-pve)The ceph-packages are all nautilus:
Bash:
~# dpkg -l | grep ceph
ii ceph 14.2.4-pve1 amd64 distributed storage and file system
ii ceph-base 14.2.4-pve1 amd64 common ceph daemon libraries and management tools
ii ceph-common 14.2.4-pve1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 14.2.4-pve1 amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 14.2.4-pve1 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 14.2.4-pve1 amd64 manager for the ceph distributed storage system
ii ceph-mon 14.2.4-pve1 amd64 monitor server for the ceph storage system
ii ceph-osd 14.2.4-pve1 amd64 OSD server for the ceph storage system
ii libcephfs2 14.2.4-pve1 amd64 Ceph distributed file system client library
ii python-ceph-argparse 14.2.4-pve1 all Python 2 utility libraries for Ceph CLI
ii python-cephfs 14.2.4-pve1 amd64 Python 2 libraries for the Ceph libcephfs libraryNow, in order to use the balancer for ceph (https://docs.ceph.com/docs/mimic/mgr/balancer), I need to set the minimum client allowed for my cluster to luminous:
Bash:
~# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 3 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anywayWhat am I missing? The only clients for this cluster are the VMs using it for their disks.
Not only VMs as client: I mount a cephfs as well (via cluster storage) and if I disable and unmount those, all clients are (as shown by running
ceph features listed as luminous.Now the big question: Can I upgrade the cluster requirements to luminous and still somehow use cephfs?
@Waschbüsch, with the update, no services (or mounted CephFS) are restarted. Once all service are restarted, there should be no Jewel client around anymore.
I have rebooted each node at least once since the upgrade. Still, the cephfs mounts are seen as done by a jewel client.
Also, deleting the cephfs storage from the cluster and recreating it does not change this behavior either.
What is weird, though: With the cephfs removed, I could carry out
Bash:
~# ceph osd set-require-min-compat-client luminous
set require_min_compat_client to luminouswithout problems. I could then re-enable the cephfs mounts. I did not get any error or warning. However, the clients are still shown as jewel:
Bash:
~# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"mds": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"osd": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 16
}
],
"client": [
{
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 3
},
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 63
}
],
"mgr": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
]
}Kernel or fuse mount? I suppose the Kernel client is still Jewel. Try to run the CephFS with fuse.
I used the UI to set up the cephfs storage. There is no mention of kernel vs. fuse in the UI? (And not mentioned in the help page either)
I manually set fuse to 1 in storage.cfg and remounted. Now every client is shown as luminous.
Thanks!
Ahm, yes. This option is not visible on the GUI.
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_cephfs
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_cephfs