(SOLVED) CEHPX Disabled, CEPH Key Migration - Removing an auth entry?

M

MN7675

Renowned Member
Dec 22, 2015
122
8
83
41
Hello, I have cephx disabled for our cluster, but I am still getting these insecure keyring errors, which I would like to migrate.

I am running into 2 issues, one is a WARN the other is a FAIL:

Code: 
WARN: No OSD in this cluster carries the fsid of client.osd-lockbox.1f256c23-4481-4bb3-93f4-40a69ffeb296, so these lockbox keys are left alone. Remove one only after confirming the OSD was destroyed and no node carries its ceph-volume device; absence from the OSD map alone is not enough.
FAIL: Ceph reports insecure keys for mds.VMHost2, mds.mdsVMHost2, mds.mdsVMHost3, mds.mdsVMHost4, mgr.VMHost3, mgr.vmhost3, but no running or configured daemon claims them, so there is no keyring to update. Before removing an auth entry, verify that the daemon was removed and no node retains its data directory or keyring.

Vmhosts2, 3 and 4 are no longer in production. I used the built-in tools to remove these hosts, properly I am very sure, but these old keys are still somewhere on the system.

Where can I find them and how can I remove them?

Also I don't know what the WARN about client.osd-lockbox is about, and whether I can clean that up as well.

Any help would be greatly appreciated.


ceph -s

Code: 
  cluster:
    id:     54da8900-a9db-4a57-923c-a62dbec8c82a
    health: HEALTH_ERR
            10 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
            Monitors are configured to allow creation of insecure key types
            4 rotating auth service keys using insecure key types
            97 auth service entities with insecure key types
            Monitors are configured to issue insecure service tickets

  services:
    mon: 3 daemons, quorum vmhost6,vmhost7,vmhost8 (age 62m) [leader: vmhost6]
    mgr: vmhost8(active, since 66m), standbys: vmhost7, vmhost6
    mds: 1/1 daemons up, 2 standby
    osd: 85 osds: 85 up (since 30m), 85 in (since 7w)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 3105 pgs
    objects: 26.55M objects, 81 TiB
    usage:   287 TiB used, 645 TiB / 932 TiB avail
    pgs:     2973 active+clean
             91   active+clean+scrubbing+deep
             41   active+clean+scrubbing

  io:
    client:   2.7 MiB/s rd, 13 MiB/s wr, 632 op/s rd, 68 op/s wr

pveversion- v

Code: 
proxmox-ve: 9.2.0 (running kernel: 7.0.14-16-pve)
pve-manager: 9.2.18 (running version: 9.2.18/614bede5d65599c6)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.14-16
proxmox-kernel-7.0.14-16-pve-signed: 7.0.14-16
proxmox-kernel-7.0.14-6-pve-signed: 7.0.14-6
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
proxmox-kernel-6.17.13-19-pve-signed: 6.17.13-19
proxmox-kernel-6.17.13-11-pve-signed: 6.17.13-11
proxmox-kernel-6.17.13-1-pve-signed: 6.17.13-1
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ceph: 20.2.4-pve4
ceph-fuse: 20.2.4-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.1
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.5-1
proxmox-backup-file-restore: 4.2.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.8
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.10
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.5
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.7
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1
 
Ah that was helpful, thank you.

Now that I understood what tool to use, I was able to delete the keys using ceph auth del <key> for all of the offending keys.

Now a new error has cropped up when doing the dry run:

Code: 
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

These are monitors that are active.

I don't know how to resolve this issue either.
 
I found this section of the ceph documentation, but I'm not sure these are the right commands to run for this problem:

Link to Documentation.

The mon. secret key is stored a keyring file in the mon data directory. It can be generated with a command like:

ceph-authtool --create-keyring /path/to/keyring --gen-key -n mon.

When creating a new monitor cluster, the keyring should also contain a client.admin key that can be used to administer the system:

ceph-authtool /path/to/keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
The resulting keyring is fed to ceph-mon --mkfs with the --keyring <keyring> command-line argument.
Click to expand...
 
Mihai said:
Ah that was helpful, thank you.

Now that I understood what tool to use, I was able to delete the keys using ceph auth del <key> for all of the offending keys.

Now a new error has cropped up when doing the dry run:

Code: 
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

These are monitors that are active.

I don't know how to resolve this issue either.
Click to expand...

Did you manage to workaround this?
 
Vmhosts2, 3 and 4 are no longer in production. I used the built-in tools to remove these hosts, properly I am very sure, but these old keys are still somewhere on the system.
Click to expand...

auth entities are per daemon. and pve does remove a daemon's entity when destroying it. Do you remember destroying each of those individually, or were the hosts removed with services still on them?

Either way `ceph auth del` is the right move at this moment.

Also I don't know what the WARN about client.osd-lockbox is about,
Click to expand...

this i1f256c23-4481-4bb3-93f4-40a69ffeb296s what ceph-volume uses to fetch the LUKS passphrase for an encrypted OSD. i think it reveals a bug, where pve-manager fails to destroy an OSD in the proper way. created a fix at https://lore.proxmox.com/pve-devel/20260920083724.2054057-1-k.chai@proxmox.com/T/#u .

you could drop it using

Bash: 
ceph auth del client.osd-lockbox.1f256c23-4481-4bb3-93f4-40a69ffeb296

if you wanted to be safe, you could double check before deleting it, confirm on every node that no ceph-volume device still carries that fsid, like:

Bash: 
ceph-volume lvm list | grep -i 1f256c23-4481-4bb3-93f4-40a69ffeb296

Regarding why ceph complains after disabling cephx: ceph's health checks are not tied to whether cephx is enabled or not. instead, the monitor reads

- allowed ciphers from the monmap,
- the key types from the auth database in it.
- and the rotating service keys

when checking the auth related healthy status.
 
  • Like
Reactions: MN7675, ahayes and SteveITS
@tchaikov

I was running into this issue:

Code: 
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

I generated these keys using these commands (one for each host):

Code: 
ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-vmhost6/keyring --gen-key -n mon.
chown ceph:ceph /var/lib/ceph/mon/ceph-vmhost6/keyring
chmod 600 /var/lib/ceph/mon/ceph-vmhost6/keyring

The dry run (/var/lib/ceph/mon/ceph-vmhost6# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys) showed that there are no issue, however, the apply stage of the command errored out:

Code: 
Rotating the shared monitor key
FAIL: error with 'auth get': mon_cmd failed - failed to find mon. in keyring

I confirmed that the file at /var/lib/ceph/mon/ceph-vmhost6/keyring contains the following content:

Code: 
[mon.]


key = Hidden-Key-Here-ADGFAST3$ADGASGDSGETC=

Same with vmhost7 and vmhost8.
 
Last edited:
Mihai said:
@tchaikov

I was running into this issue:

Code: 
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

I generated these keys using these commands (one for each host):

Code: 
ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-vmhost6/keyring --gen-key -n mon.
chown ceph:ceph /var/lib/ceph/mon/ceph-vmhost6/keyring
chmod 600 /var/lib/ceph/mon/ceph-vmhost6/keyring

The dry run (/var/lib/ceph/mon/ceph-vmhost6# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys) showed that there are no issue, however, the apply stage of the command errored out:

Code: 
Rotating the shared monitor key
FAIL: error with 'auth get': mon_cmd failed - failed to find mon. in keyring

I confirmed that the file at /var/lib/ceph/mon/ceph-vmhost6/keyring contains the following content:

Code: 
[mon.]


key = Hidden-Key-Here-ADGFAST3$ADGASGDSGETC=

Same with vmhost7 and vmhost8.
Click to expand...
I have the exact same situation. Advice on how to proceed is highly appreciated.
 
FWIW: If it were necessary to first re-enable cephx on the cluster to proceed with migration, steps on how to do that *safely* (and hopefully without downtime?) would be appreciated.
 
An update on my situation. Running ceph auth ls shows that there are no entries for mon keys, but they do exist on file.

I must be missing a step to somehow enter the generated mon keys into some sort of ceph database?

I just don't know what ceph-authtool command to run and I'm anxious about causing more problems.
 
I solved this for my cluster. The steps were thus:

Make sure all your monitor keyring files are identical (all mon use the same key!) and of this structure (use your real client-admin key)

Code: 
[mon.]
        key = <somekey>
        caps mon = "allow *"
[client.admin]
        key = <anotherkey>
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

then, once(!)
run (eg. on node1):

ceph auth import -i /var/lib/ceph/mon/ceph-node1/keyring

now running ceph auth ls should show the mon key.

for me that was enough to complete the process running the

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys

tool once again.
 
Last edited:
  • Like
Reactions: MN7675
Waschbüsch said:
I solved this for my cluster. The steps were thus:

Make sure all your monitor keyring files are identical (all mon use the same key!) and of this structure (use your real client-admin key)

Code: 
[mon.]
        key = <somekey>
        caps mon = "allow *"
[client.admin]
        key = <anotherkey>
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

then, once(!)
run (eg. on node1):

ceph auth import -i /var/lib/ceph/mon/ceph-node1/keyring

now running ceph auth ls should show the mon key.

for me that was enough to complete the process running the

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys

tool once again.
Click to expand...

Wow what a mess.

I had to manually refresh a client key:

Code: 
# Export the current keyring (includes key and caps)
ceph auth get client.nvr02 -o /tmp/myapp.keyring

# Regenerate the key in the keyring file
ceph-authtool /tmp/myapp.keyring -n client.nvr02 --gen-key

# Import the updated keyring back into the cluster
ceph auth import -i /tmp/myapp.keyring

And I was finally able to restrict ciphers.

I'm still waiting on this to automatically clear, I hope:

Code: 
HEALTH_WARN: 4 rotating auth service keys using insecure key types
This usually clears within a few hours after the monitors start issuing aes256k service tickets.

rotating service keys for mon using insecure key type: aes
rotating service keys for mds using insecure key type: aes
rotating service keys for osd using insecure key type: aes
rotating service keys for mgr using insecure key type: aes
 
Waschbüsch said:
I solved this for my cluster. The steps were thus:

Make sure all your monitor keyring files are identical (all mon use the same key!) and of this structure (use your real client-admin key)

Code: 
[mon.]
        key = <somekey>
        caps mon = "allow *"
[client.admin]
        key = <anotherkey>
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

then, once(!)
run (eg. on node1):

ceph auth import -i /var/lib/ceph/mon/ceph-node1/keyring

now running ceph auth ls should show the mon key.

for me that was enough to complete the process running the

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys

tool once again.
Click to expand...

Wow okay I made significant progress.

There is a single keyring for a client which I had to manually refresh I guess.

Now I am having a NEW Issue.

None of the Windows guests can start anymore. The TPM State disks cannot be mapped:

Code: 
task started by HA resource agent
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: sysfs write failed
TASK ERROR: start failed: can't map rbd volume vm-116-disk-2: rbd: sysfs write failed

running dmesg tail shows:

Code: 
[ 2977.077854] libceph: auth protocol 'cephx' not allowed
[ 3022.519362] libceph: auth protocol 'cephx' not allowed
[ 3032.213888] libceph: auth protocol 'cephx' not allowed
 
Do you have a keyring in /etc/pve/priv/ceph/<your_rbd_storage>.keyring?
If so, AND you still have
Code: 
    auth_client_required = none
    auth_cluster_required = none
    auth_service_required = none
set in your ceph.conf, try renaming that to <your_rbd_storage>.keyring_old (and have the reverse command handy in case it makes things worse).
I have found that if the key is defined, some parts of proxmox seem to want to use authentication where none is required.

I had to do that once backup attempts failed with a "cannot connect to cluster" error...
 
Last edited:
  • Like
Reactions: MN7675
Here's the contents of my
Waschbüsch said:
Do you have a keyring in /etc/pve/priv/ceph/<your_rbd_storage>.keyring?
If so, AND you still have
Code: 
    auth_client_required = none
    auth_cluster_required = none
    auth_service_required = none
set in your ceph.conf, try renaming that to <your_rbd_storage>.keyring_old (and have the reverse command handy in case it makes things worse).
I have found that if the key is defined, some parts of proxmox seem to want to use authentication where none is required.

I had to do that once backup attempts failed with a "cannot connect to cluster" error...
Click to expand...

You are a lifesaver. That was exactly it.
 
After cleaning all the open sessions and running the last step i got this error.
Bash: 
~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
FAIL: error with 'auth commit-pending': mon_cmd failed - creating key with insecure key type ("`preferred' AKA aes") not allowed

Running the following command let me run the script:
Bash: 
ceph mon set auth_preferred_cipher aes256k

Maybe this may help people with the same error.
 
  • Like
Reactions: MN7675
You must log in or register to reply here.
Top Bottom
Back