Cannot mount CephFS filesystems on VMs or external systems after upgrading to Ceph 20.2.4

[mlazarin]

Hi! I'm facing an issue with mounting CephFS filesystems after the upgrade to latest version of Ceph Tentacle (20.2.4). From Proxmox perspective everything seems fine (on the UI and also manually):

 root on vmserver01: /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
 root on vmserver01: ceph -s
  cluster:
    id:     e3ABCD3a-XXXX-YYYY-ZZZZ-70bdABCDe2b8 # (some characters redacted for privacy)
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum vmserver01,vmserver02,vmserver03 (age 2h) [leader: vmserver01]
    mgr: vmserver02(active, since 47h), standbys: vmserver03, vmserver01
    mds: 2/2 daemons up, 4 standby
    osd: 18 osds: 18 up (since 6h), 18 in (since 4d)

  data:
    volumes: 2/2 healthy
    pools:   13 pools, 865 pgs
    objects: 55.97k objects, 200 GiB
    usage:   546 GiB used, 16 TiB / 17 TiB avail
    pgs:     865 active+clean

  io:
    client:   0 B/s rd, 32 KiB/s wr, 0 op/s rd, 5 op/s wr

 root on vmserver01: ceph -v
ceph version 20.2.4 (4302593b7d5304637a99b628879b32886b798252) tentacle (stable)
 root on vmserver01: ceph fs ls
name: cephfs_vm, metadata pool: cephfs_vm_metadata, data pools: [cephfs_vm_data ]
name: cephfs_ctnrs, metadata pool: cephfs_ctnrs_metadata, data pools: [cephfs_ctnrs_data ]
 root on vmserver01: ceph health detail
HEALTH_OK
 root on vmserver01:

After that I recreated the users for CephFS access (I use separate users for each CephFS I have):

ceph auth del client.cephfs_vm_user
ceph auth ls | grep cephfs_vm_user
ceph auth get-or-create client.cephfs_vm_user \
  mds "allow rw fsname=cephfs_vm" \
  mgr "allow rw" \
  mon "allow r fsname=cephfs_vm" \
  osd "allow rw tag cephfs data=cephfs_vm"

I got the new generated keys, and updated the keyring files on my Linux desktop, to mount again the existing CephFS. But when I try to mount, I'm getting the following error message (I'm showing the manual try to mount, but via /etc/fstab I get the same error):

 root on dsktplin: mount -t ceph 10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789:/ /mnt/test -o name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.cephfs_vm_user.secret,fs=cephfs_vm
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
 root on dsktplin:

Additional information:
Version numbers:
Proxmox VE: 9.2.18
Ceph on Proxmox: 20.2.4
Desktop Ubuntu: 26.04 (latest updates)
Ceph on Ubuntu: 20.2.0 (latest updates)

Information about the keyring keys:
On previous versions of Ceph: 40 characters long, format AQ...== (finishing with double equal signs - example "AQBjfL9pJ2GBLhAA1xAogEzsEIzXoECSuo7cHA==")
On newest version of Ceph: 60 characters long, format Ag...= (yes, finishing with a single equal sign - example "AgB686VqXm3kFCAAwBZaL1PL9c4F7KipNyauBCj//ZjP2czuw94nW3XzIb4=")

Is anyone facing the same issues? Any ideas on how to fix this?

 

[fabian]

could you verify that your kernel version on the external clients is new enough (>= 7.0), and that the secret is correct there as well? are there any more related messages in the external clients journal?

 

[mlazarin]

Hi! Yes, I'm using the latest 7.0 kernel (Ubuntu 26.04), and didn't find any logs on the journal (maybe I'm looking in the wrong way, if you could please provide more details I'll check again). I tried the mount with more verbose output, and that's what I got:

 root on dsktplin: mount -v -t ceph 10.25.31.10:6789,10.25.31.20:6789,10.25.31.30:6789:/ /mnt/test -o name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.ceph
fs_vm_user.secret,fs=cephfs_vm
parsing options: rw,name=cephfs_vm_user,secretfile=/etc/ceph/ceph.client.cephfs_vm_user.secret,fs=cephfs_vm
mount.ceph: options "name=cephfs_vm_user,mds_namespace=cephfs_vm".
invalid new device string format
Could not discover monitor addresses
keyring.get_secret failed
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument
couldn't append secret option: -22
 root on dsktplin: uname -a
Linux dsktplin 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug  1 04:26:38 UTC 2026 x86_64 GNU/Linux
 root on dsktplin:

Any other ideas? Do you see any errors on the mount command? It's the same I was using before the upgrade (maybe something changed?)