CEHPX Disabled, CEPH Key Migration - Removing an auth entry?

[Mihai]

Hello, I have cephx disabled for our cluster, but I am still getting these insecure keyring errors, which I would like to migrate.

I am running into 2 issues, one is a WARN the other is a FAIL:

WARN: No OSD in this cluster carries the fsid of client.osd-lockbox.1f256c23-4481-4bb3-93f4-40a69ffeb296, so these lockbox keys are left alone. Remove one only after confirming the OSD was destroyed and no node carries its ceph-volume device; absence from the OSD map alone is not enough.
FAIL: Ceph reports insecure keys for mds.VMHost2, mds.mdsVMHost2, mds.mdsVMHost3, mds.mdsVMHost4, mgr.VMHost3, mgr.vmhost3, but no running or configured daemon claims them, so there is no keyring to update. Before removing an auth entry, verify that the daemon was removed and no node retains its data directory or keyring.

Vmhosts2, 3 and 4 are no longer in production. I used the built-in tools to remove these hosts, properly I am very sure, but these old keys are still somewhere on the system.

Where can I find them and how can I remove them?

Also I don't know what the WARN about client.osd-lockbox is about, and whether I can clean that up as well.

Any help would be greatly appreciated.


ceph -s

  cluster:
    id:     54da8900-a9db-4a57-923c-a62dbec8c82a
    health: HEALTH_ERR
            10 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
            Monitors are configured to allow creation of insecure key types
            4 rotating auth service keys using insecure key types
            97 auth service entities with insecure key types
            Monitors are configured to issue insecure service tickets

  services:
    mon: 3 daemons, quorum vmhost6,vmhost7,vmhost8 (age 62m) [leader: vmhost6]
    mgr: vmhost8(active, since 66m), standbys: vmhost7, vmhost6
    mds: 1/1 daemons up, 2 standby
    osd: 85 osds: 85 up (since 30m), 85 in (since 7w)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 3105 pgs
    objects: 26.55M objects, 81 TiB
    usage:   287 TiB used, 645 TiB / 932 TiB avail
    pgs:     2973 active+clean
             91   active+clean+scrubbing+deep
             41   active+clean+scrubbing

  io:
    client:   2.7 MiB/s rd, 13 MiB/s wr, 632 op/s rd, 68 op/s wr

pveversion- v

proxmox-ve: 9.2.0 (running kernel: 7.0.14-16-pve)
pve-manager: 9.2.18 (running version: 9.2.18/614bede5d65599c6)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.14-16
proxmox-kernel-7.0.14-16-pve-signed: 7.0.14-16
proxmox-kernel-7.0.14-6-pve-signed: 7.0.14-6
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
proxmox-kernel-6.17.13-19-pve-signed: 6.17.13-19
proxmox-kernel-6.17.13-11-pve-signed: 6.17.13-11
proxmox-kernel-6.17.13-1-pve-signed: 6.17.13-1
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ceph: 20.2.4-pve4
ceph-fuse: 20.2.4-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.1
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.5-1
proxmox-backup-file-restore: 4.2.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.8
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.10
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.5
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.7
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1

 

[gurubert]

What does "ceph auth ls" show? Do not post the output here but look for the mentioned keys. If not needed any more they can be deleted.

 

[Mihai]

Ah that was helpful, thank you.

Now that I understood what tool to use, I was able to delete the keys using ceph auth del <key> for all of the offending keys.

Now a new error has cropped up when doing the dry run:

FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

These are monitors that are active.

I don't know how to resolve this issue either.

 

[Mihai]

I found this section of the ceph documentation, but I'm not sure these are the right commands to run for this problem:

Link to Documentation.

The mon. secret key is stored a keyring file in the mon data directory. It can be generated with a command like:

ceph-authtool --create-keyring /path/to/keyring --gen-key -n mon.

When creating a new monitor cluster, the keyring should also contain a client.admin key that can be used to administer the system:

ceph-authtool /path/to/keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
The resulting keyring is fed to ceph-mon --mkfs with the --keyring <keyring> command-line argument.

 

[longer]

Ah that was helpful, thank you.

Now that I understood what tool to use, I was able to delete the keys using ceph auth del <key> for all of the offending keys.

Now a new error has cropped up when doing the dry run:

FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

These are monitors that are active.

I don't know how to resolve this issue either.

Did you manage to workaround this?

 

[tchaikov]

Vmhosts2, 3 and 4 are no longer in production. I used the built-in tools to remove these hosts, properly I am very sure, but these old keys are still somewhere on the system.

auth entities are per daemon. and pve does remove a daemon's entity when destroying it. Do you remember destroying each of those individually, or were the hosts removed with services still on them?

Either way `ceph auth del` is the right move at this moment.

Also I don't know what the WARN about client.osd-lockbox is about,

this i1f256c23-4481-4bb3-93f4-40a69ffeb296s what ceph-volume uses to fetch the LUKS passphrase for an encrypted OSD. i think it reveals a bug, where pve-manager fails to destroy an OSD in the proper way. created a fix at https://lore.proxmox.com/pve-devel/20260920083724.2054057-1-k.chai@proxmox.com/T/#u .

you could drop it using

ceph auth del client.osd-lockbox.1f256c23-4481-4bb3-93f4-40a69ffeb296

if you wanted to be safe, you could double check before deleting it, confirm on every node that no ceph-volume device still carries that fsid, like:

ceph-volume lvm list | grep -i 1f256c23-4481-4bb3-93f4-40a69ffeb296

Regarding why ceph complains after disabling cephx: ceph's health checks are not tied to whether cephx is enabled or not. instead, the monitor reads

- allowed ciphers from the monmap,
- the key types from the auth database in it.
- and the rotating service keys

when checking the auth related healthy status.