Proxmox as CEPH Nautilus client

This depends on the 'ceph osd set-require-min-compat-client' setting and defines the oldest client that can connect. Here you have to keep the kernel client in mind too, so luminous or jewel if stock debian ceph packages are used (fresh install).
 
Currently we have mimic on CentOS, min client is jewel. So, theoritically ceph client in proxmox can connect to nautilus ceph, if it allow it.

thanks.
 
Yes and easy to try. ;)
 
With 6.0-4 released what is the roadmap to migrate the Ceph client to a newer version?
 
I have 5.4 VE connected to external Nautilus cluster. Not seen any problem yet.
I have no issues getting to connect to Nautilus. It appears that Proxmox is still using the Jewel client which limits the ability to use some settings.
 
https://pve.proxmox.com/wiki/Package_Repositories

This is Proxmox VE’s main Ceph repository and holds the Ceph packages for production use. You can also use this repository to update only the Ceph client.

File /etc/apt/sources.list.d/ceph.list
deb http://download.proxmox.com/debian/ceph-nautilus buster main
It would appear that 6.0-4 is still using the jewel client.

Error EPERM: cannot set require_min_compat_client to luminous: 3 connected client(s) look like jewel (missing 0x800000000000000);
 
It would appear that 6.0-4 is still using the jewel client.
A fresh Proxmox VE 6 or buster installation uses the luminous client.
Code:
:~# dpkg -l | grep ceph
ii  ceph-common                          12.2.11+dfsg1-2.1           amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            12.2.11+dfsg1-2.1           amd64        FUSE-based client for the Ceph distributed file system
ii  libcephfs2:amd64                     12.2.11+dfsg1-2.1           amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.11+dfsg1-2.1           amd64        Python libraries for the Ceph libcephfs library
 
A fresh Proxmox VE 6 or buster installation uses the luminous client.
Code:
:~# dpkg -l | grep ceph
ii  ceph-common                          12.2.11+dfsg1-2.1           amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            12.2.11+dfsg1-2.1           amd64        FUSE-based client for the Ceph distributed file system
ii  libcephfs2:amd64                     12.2.11+dfsg1-2.1           amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.11+dfsg1-2.1           amd64        Python libraries for the Ceph libcephfs library

Odd I show the file versions as 14.2.1-pve2 for these files.

I installed 5.4 Fresh with no Ceph upgraded to 6.0-1 Beta install Ceph and then upgraded to 6.0-4 release
 
I installed 5.4 Fresh with no Ceph upgraded to 6.0-1 Beta install Ceph and then upgraded to 6.0-4 release
Once you installed ceph explicitly, then it is 14.2.1. The packages I mentioned are from a stock install, without any extra package installation.
 
There is still something amiss. I have upgraded my cluster to

Bash:
~# pveversion
pve-manager/6.0-9/508dcee0 (running kernel: 5.0.21-3-pve)

The ceph-packages are all nautilus:

Bash:
~# dpkg -l | grep ceph
ii  ceph                                 14.2.4-pve1                             amd64        distributed storage and file system
ii  ceph-base                            14.2.4-pve1                             amd64        common ceph daemon libraries and management tools
ii  ceph-common                          14.2.4-pve1                             amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            14.2.4-pve1                             amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             14.2.4-pve1                             amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             14.2.4-pve1                             amd64        manager for the ceph distributed storage system
ii  ceph-mon                             14.2.4-pve1                             amd64        monitor server for the ceph storage system
ii  ceph-osd                             14.2.4-pve1                             amd64        OSD server for the ceph storage system
ii  libcephfs2                           14.2.4-pve1                             amd64        Ceph distributed file system client library
ii  python-ceph-argparse                 14.2.4-pve1                             all          Python 2 utility libraries for Ceph CLI
ii  python-cephfs                        14.2.4-pve1                             amd64        Python 2 libraries for the Ceph libcephfs library

Now, in order to use the balancer for ceph (https://docs.ceph.com/docs/mimic/mgr/balancer), I need to set the minimum client allowed for my cluster to luminous:

Bash:
~# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 3 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway

What am I missing? The only clients for this cluster are the VMs using it for their disks.
 
What am I missing? The only clients for this cluster are the VMs using it for their disks.

Not only VMs as client: I mount a cephfs as well (via cluster storage) and if I disable and unmount those, all clients are (as shown by running ceph features listed as luminous.

Now the big question: Can I upgrade the cluster requirements to luminous and still somehow use cephfs?
 
@Waschbüsch, with the update, no services (or mounted CephFS) are restarted. Once all service are restarted, there should be no Jewel client around anymore.
 
@Waschbüsch, with the update, no services (or mounted CephFS) are restarted. Once all service are restarted, there should be no Jewel client around anymore.

I have rebooted each node at least once since the upgrade. Still, the cephfs mounts are seen as done by a jewel client.
Also, deleting the cephfs storage from the cluster and recreating it does not change this behavior either.

What is weird, though: With the cephfs removed, I could carry out

Bash:
~# ceph osd set-require-min-compat-client luminous
set require_min_compat_client to luminous

without problems. I could then re-enable the cephfs mounts. I did not get any error or warning. However, the clients are still shown as jewel:

Bash:
~# ceph features
{
    "mon": [
        {
            "features": "0x3ffddff8ffacffff",
            "release": "luminous",
            "num": 3
        }
    ],
    "mds": [
        {
            "features": "0x3ffddff8ffacffff",
            "release": "luminous",
            "num": 3
        }
    ],
    "osd": [
        {
            "features": "0x3ffddff8ffacffff",
            "release": "luminous",
            "num": 16
        }
    ],
    "client": [
        {
            "features": "0x27018fb86aa42ada",
            "release": "jewel",
            "num": 3
        },
        {
            "features": "0x3ffddff8ffacffff",
            "release": "luminous",
            "num": 63
        }
    ],
    "mgr": [
        {
            "features": "0x3ffddff8ffacffff",
            "release": "luminous",
            "num": 3
        }
    ]
}
 
till, the cephfs mounts are seen as done by a jewel client.
Kernel or fuse mount? I suppose the Kernel client is still Jewel. Try to run the CephFS with fuse.
 
Kernel or fuse mount? I suppose the Kernel client is still Jewel. Try to run the CephFS with fuse.

I used the UI to set up the cephfs storage. There is no mention of kernel vs. fuse in the UI? (And not mentioned in the help page either)

I manually set fuse to 1 in storage.cfg and remounted. Now every client is shown as luminous.

Thanks!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!