Cannot mount CephFS filesystems on VMs or external systems after upgrading to Ceph 20.2.4

M

mlazarin

Member
Dec 10, 2023
8
1
8
Hi! I'm facing an issue with mounting CephFS filesystems after the upgrade to latest version of Ceph Tentacle (20.2.4). From Proxmox perspective everything seems fine (on the UI and also manually):

Code: 
 root on vmserver01: /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
 root on vmserver01: ceph -s
  cluster:
    id:     e3ABCD3a-XXXX-YYYY-ZZZZ-70bdABCDe2b8 # (some characters redacted for privacy)
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum vmserver01,vmserver02,vmserver03 (age 2h) [leader: vmserver01]
    mgr: vmserver02(active, since 47h), standbys: vmserver03, vmserver01
    mds: 2/2 daemons up, 4 standby
    osd: 18 osds: 18 up (since 6h), 18 in (since 4d)

  data:
    volumes: 2/2 healthy
    pools:   13 pools, 865 pgs
    objects: 55.97k objects, 200 GiB
    usage:   546 GiB used, 16 TiB / 17 TiB avail
    pgs:     865 active+clean

  io:
    client:   0 B/s rd, 32 KiB/s wr, 0 op/s rd, 5 op/s wr

 root on vmserver01: ceph -v
ceph version 20.2.4 (4302593b7d5304637a99b628879b32886b798252) tentacle (stable)
 root on vmserver01: ceph fs ls
name: cephfs_vm, metadata pool: cephfs_vm_metadata, data pools: [cephfs_vm_data ]
name: cephfs_ctnrs, metadata pool: cephfs_ctnrs_metadata, data pools: [cephfs_ctnrs_data ]
 root on vmserver01: ceph health detail
HEALTH_OK
 root on vmserver01:

After that I recreated the users for CephFS access (I use separate users for each CephFS I have):

Code: 
ceph auth del client.cephfs_vm_user
ceph auth ls | grep cephfs_vm_user
ceph auth get-or-create client.cephfs_vm_user \
  mds "allow rw fsname=cephfs_vm" \
  mgr "allow rw" \
  mon "allow r fsname=cephfs_vm" \
  osd "allow rw tag cephfs data=cephfs_vm"

I got the new generated keys, and updated the keyring files on my Linux desktop, to mount again the existing CephFS. But when I try to mount, I'm getting the following error message (I'm showing the manual try to mount, but via /etc/fstab I get the same error):

Code: 
 root on dsktplin: mount -t ceph 10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789:/ /mnt/test -o name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.cephfs_vm_user.secret,fs=cephfs_vm
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
 root on dsktplin:

Additional information:
Version numbers:
Proxmox VE: 9.2.18
Ceph on Proxmox: 20.2.4
Desktop Ubuntu: 26.04 (latest updates)
Ceph on Ubuntu: 20.2.0 (latest updates)

Information about the keyring keys:
On previous versions of Ceph: 40 characters long, format AQ...== (finishing with double equal signs - example "AQBjfL9pJ2GBLhAA1xAogEzsEIzXoECSuo7cHA==")
On newest version of Ceph: 60 characters long, format Ag...= (yes, finishing with a single equal sign - example "AgB686VqXm3kFCAAwBZaL1PL9c4F7KipNyauBCj//ZjP2czuw94nW3XzIb4=")

Is anyone facing the same issues? Any ideas on how to fix this?
 
could you verify that your kernel version on the external clients is new enough (>= 7.0), and that the secret is correct there as well? are there any more related messages in the external clients journal?
 
Hi! Yes, I'm using the latest 7.0 kernel (Ubuntu 26.04), and didn't find any logs on the journal (maybe I'm looking in the wrong way, if you could please provide more details I'll check again). I tried the mount with more verbose output, and that's what I got:

Code: 
 root on dsktplin: mount -v -t ceph 10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789:/ /mnt/test -o name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.ceph
fs_vm_user.secret,fs=cephfs_vm
parsing options: rw,name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.cephfs_vm_user.secret,fs=cephfs_vm
mount.ceph: options "name=cephfs_vm_user,mds_namespace=cephfs_vm".
invalid new device string format
Could not discover monitor addresses
keyring.get_secret failed
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
 root on dsktplin: uname -a
Linux dsktplin 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug  1 04:26:38 UTC 2026 x86_64 GNU/Linux
 root on dsktplin:

Any other ideas? Do you see any errors on the mount command? It's the same I was using before the upgrade (maybe something changed?)
 
Seems the syntax changed according to latest documentation, but even with the updated syntax, it's still failing:

Code: 
 root on dsktplin: mount -v -t ceph vm_test@.cephfs_vm=/ /mnt/test -o mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
parsing options: rw,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
keyring.from_ceph_context failed: -5
2026-09-15T11:17:11.813-0500 7ce4214a8080 -1 auth: error parsing file /etc/ceph/ceph.client.vm_test.keyring: error setting modifier for [client.vm_test] type=key val=Ag<PARTIALLY REDACTED>BA=: Malformed input [buffer:3]
2026-09-15T11:17:11.813-0500 7ce4214a8080 -1 auth: failed to load /etc/ceph/ceph.client.vm_test.keyring: (5) Input/output error
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
 root on dsktplin:

I did the test above generating a new user/key from scratch, as per Ceph mount guide: https://docs.ceph.com/en/latest/cephfs/mount-using-kernel-driver/

I triple checked the values of the generated key, the file content, and even the <PARTIALLY REDACTED> key from above log, and they all match. Seems really that the mount helper is not able to properly send the key to the Linux Kernel (the complain about the not valid base64), even if the kernel has the minimal requirement (>=7.0).

The only other point I can think about is that Ceph in the cluster is V20.2.4, and the Ceph client in Ubuntu 26.04 is V20.2.0, but there is no Ceph repository for latest debian distributions (Ubuntu 26.04 Resolute Raccoon or Debian 13 Trixie) to move to a newer version, and V20.2.0 in Ubuntu 26.04 is the latest one coming from Ubuntu.

I'll try again later using Ubuntu 24.04 in a VM, with the latest Ceph client (seems we have a Ceph tentacle repository for Ubuntu 24.04, so I'll try to dowload the latest client over the Ubuntu 24.04 standard version to see if it works).

Any other ideas/comments are really appreciated!
 
IMPORTANT UPDATE: I did some tests with Fedora 44 and Ubuntu 24.04 + HWE, and below I'm documenting this so it can help others. I could mount with Fedora 44 and Ubuntu 24.04 + HWE, but still not possible on Ubuntu 26.04.

I created a VM using Fedora 44 (latest server ISO downloaded from https://fedoraproject.org/server/), and copied the exact same ceph.conf, ceph.client.vm_test.keyring, ceph.client.vm_test.secret files that I used to test on Ubuntu 26.04 before (and that failed as per the previous post).

After default installation of Fedora 44, I installed ceph-common using the default Fedora repositories (dnf install ceph-common), and got the following:

Code: 
root@localhost:~# dnf install ceph-common
Updating and loading repositories:
 Fedora 44 openh264 (From Cisco) - x86_64                                       100% |   3.4 KiB/s |   5.3 KiB |  00m02s
 Fedora 44 - x86_64 - Updates                                                   100% |   3.5 MiB/s |  11.8 MiB |  00m03s
 Fedora 44 - x86_64                                                             100% |   8.9 MiB/s |  36.6 MiB |  00m04s
Repositories loaded.
Package                                 Arch       Version                                 Repository               Size
Upgrading:
 expat                                  x86_64     0:2.8.3-1.fc44                          updates             334.4 KiB
   replacing expat                      x86_64     0:2.7.3-2.fc44                          anaconda            316.8 KiB
 libibverbs                             x86_64     0:61.1^20260812git975bdaf-1.fc44        updates               1.3 MiB
   replacing libibverbs                 x86_64     0:61.0-2.fc44                           anaconda              1.3 MiB
 rdma-core-common                       noarch     0:61.1^20260812git975bdaf-1.fc44        updates              21.9 KiB
   replacing rdma-core-common           noarch     0:61.0-2.fc44                           anaconda             21.9 KiB
Installing:
 ceph-common                            x86_64     2:20.2.4-1.fc44                         updates             109.7 MiB
... <HIDING MANY DEPENDENCIES INSTALLATION>
Complete!
root@localhost:~# ceph -v
ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)
root@localhost:~# uname -a
Linux localhost.localdomain 6.19.10-300.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 25 18:23:49 UTC 2026 x86_64 GNU/Linux
root@localhost:~#

After that I copied the 3 files mentioned (ceph.conf, ceph.client.vm_test.keyring, ceph.client.vm_test.secret) to /etc/ceph and tried to mount my CephFS from my Proxmox Ceph cluster:

Code: 
root@localhost:/etc/ceph# mount -v -t ceph vm_test@.cephfs_vm=/ /mnt/test -o mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
parsing options: rw,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
root@localhost:/etc/ceph#

After this failure (expected, since the standard kernel in Fedora 44 was 6.19), I did a full system update, using dnf update:

Code: 
root@localhost:/etc/ceph# dnf update
Updating and loading repositories:
Repositories loaded.
Package                                       Arch     Version                                 Repository           Size
Upgrading:
 NetworkManager                               x86_64   1:1.56.1-2.fc44                         updates           6.1 MiB
...
[806/806] Removing libgcc-0:16.0.1-0.10.fc44.x86_64                             100% |   0.0   B/s |  11.0   B |  02m13s
Complete!
root@localhost:/etc/ceph# reboot

After reboot:

Code: 
root@localhost:/etc/ceph# uname -a
Linux localhost.localdomain 7.2.5-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 11 15:11:05 UTC 2026 x86_64 GNU/Linux
root@localhost:/etc/ceph# ceph -v
ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)
root@localhost:/etc/ceph# mount -v -t ceph vm_test@.cephfs_vm=/ /mnt/test -o mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
parsing options: rw,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
mount.ceph: resolved to: "10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789"
mount.ceph: trying mount with new device syntax: vm_test@e3307b3a-a8cc-4b58-a697-70bd7ed1e2b8.cephfs_vm=/
mount.ceph: options "name=vm_test,key=vm_test,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789" will pass to kernel
mount: /mnt/test does not contain SELinux labels.
       You just mounted a file system that supports labels which does not
       contain labels, onto an SELinux box. It is likely that confined
       applications will generate AVC messages and not be allowed access to
       this file system.  For more details see restorecon(8) and mount(8).
root@localhost:/etc/ceph#

Success!! So the combination of the latest Ceph client (20.2.4) and the latest kernel 7.2.5 makes it work in Fedora. Performed additional tests such as folder and file creation, etc., and all worked as expected.

Then I tried to check how to bring Ceph 20.2.4 and a Kernel 7.0 to Ubuntu. Since there are no Ceph repositories to Ubuntu 26.04 (Resolute Raccoon), the only alternative was to use Ubuntu 24.04 (Noble Numbat). To have a kernel 7.0 on Ubuntu 24.04, I had to install the option HWE which allow installing kernel 7.0 on Ubuntu 24.04 LTS.

To install HEW on Ubuntu 24.04 I used the below commands:
Code: 
sudo apt update
sudo apt install --install-recommends linux-generic-hwe-24.04
reboot

After the reboot, I added Ceph repository to Ubuntu 24.04 + HWE, and updated ceph-common and ceph-fuse packages to version 20.2.4.

Then I tried to mount the CephFS filesystem using the same 3 configuration files (ceph.conf, ceph.client.vm_test.keyring, ceph.client.vm_test.secret) that I used before on Ubuntu 26.04 and Fedora 44:

Code: 
 root on technitiumtest03 at …/ceph uname -a
Linux technitiumtest03 7.0.0-31-generic #31~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Aug 10 09:38:02 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
 root on technitiumtest03 at …/ceph ceph -v
ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)
 root on technitiumtest03 at …/ceph mount -v -t ceph vm_test@.cephfs_vm=/ /mnt/test -o mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
parsing options: rw,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789,secretfile=/etc/ceph/ceph.client.vm_test.secret
mount.ceph: resolved to: "10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789"
mount.ceph: trying mount with new device syntax: vm_test@e3307b3a-a8cc-4b58-a697-70bd7ed1e2b8.cephfs_vm=/
mount.ceph: options "name=vm_test,key=vm_test,mon_addr=10.25.31.10:6789/10.25.31.20:6789/10.25.31.30:6789" will pass to kernel
 root on technitiumtest03 at …/ceph ll /mnt/test
total 4
drwxr-xr-x 4 root root    2 Sep 15 21:38 .
drwxr-xr-x 6 root root 4096 Sep 15 22:34 ..
drwxr-xr-x 2 root root    0 Mar 21 12:58 dump
drwxr-xr-x 4 root root    2 Mar 21 12:58 template
 root on technitiumtest03 at …/ceph

So success as well!!

This means that to be able to mount a CephFS filesystem from a cluster that was upgraded to Ceph 20.2.4 seems to be mandatory to use a client at least at the same level, and any kernel 7.0. In previous Ceph versions this was more flexible, and using similar versions would work (for example to connect to a cluster on Ceph 20.2.2 using a client 20.2.0). This last change which implemented aes256k force to use the same level on the client.

Important limitations I can see right now for VMs or external systems:
- Debian 13 (Trixie) does not have a respective Ceph repository that allow installation of Ceph 20.2.4
- Ubuntu 26.04 (Resolute Raccoon) does not have a respective Ceph repository that allow installation of Ceph 20.2.4
- Ubuntu 24.04 (Noble Numbat) has the respective Ceph repository that allow installation of Ceph 20.2.4, but to install kernel 7.0 needs to be extended with HWE option

On Proxmox nodes running Debian 13 with latest updates, this is covered (ceph obviously is version 20.2.4 there!).

I didn't test Debian 12/13 or other distros.

Fedora 44 and Ubuntu 24.04 + HWE with the right combination (ceph 20.2.4 and kernel 7.0) worked properly.

So if you are planning to upgrade Ceph in your Proxmox cluster and need to mount CephFS filesystems, please check if this exact combination (ceph client 20.2.4 and kernel 7.0) is available for your distro!

Hope this helps other Proxmox users!
 
My next step is to decide the future of my homelab (move to Fedora? implement HWE on all Ubuntu nodes? other options?). I'll probably move to Ubuntu 24.04 + HWE to preserve all my automations (Ansible/Terraform), downgrade my Linux Desktop back to 24.04, and wait for future Ubuntu 26.04 updates before migrating. :rolleyes::rolleyes::rolleyes:
 
this is why our migration script checks for existing incompatible clients.. did you do the migration while the old clients were offline?

The helper reports sessions that may still hold an old key. Client namesare hints, not a complete workload inventory; check disconnected clients and external key copies yourself.
Click to expand...
 
for Trixie at least, you should be able to use our Ceph repositories for PVE-9 to get current versions. for Ubuntu, those packages are untested - they might or might not be compatible.
 
Hi, Fabian. Yes, they were disconnected, I was at the middle of my homelab rebuild. Agree about using Proxmox repositories for Ceph on Debian Trixie, but that's not true for Ubuntu for example. And my point is not about the upgrade process itself, but the readiness of different distros as clients not only for today, but in short term, since some distros are focused on stability (with "older" packages, like Debian/Ubuntu) while others are focused on the latest & greatest (like Fedora). Both approaches have pros & cons, Proxmox users should evaluate this aspect before the upgrade, even for VMs or external clients that they don't have yet, or are planning to have in a short term period. In those cases they may need a different distro, a mixed approach like Ubuntu 24.04+HWE as I mentioned above, or wait. But with the information they can take a better decision under their situation (and this is for sure a case-by-case basis).
 
You must log in or register to reply here.
Top Bottom
Back