Ceph Squid to Tentacle Migration

R

RMaxwell72

New Member
Sep 14, 2026
5
0
1
Hello, the guide overall was helpful, but ran into the same issue as another user where I had to increase the timeout in the helper script to be able to update the CephFS keys:

Code: 
perl -0pi -e 's/(PVE::Tools::run_command\(\s*\[\x27umount\x27,\s*\$path\],\s*timeout\s*=>\s*)20/$1 65/' \
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys

After that the storage keys and admin key was successful:

Code: 
root@pve01:/etc/pve/priv/ceph# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys     --rotate-all-storage-keys --rotate-admin-key --apply
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
  redid the 'cephfs' mount on node 'pve01', which now reads the staged key
  redid the 'cephfs' mount on node 'pve02', which now reads the staged key
  redid the 'cephfs' mount on node 'pve03', which now reads the staged key
PASS: Every service key uses 'aes256k', and so do the service tickets.

Ready for confirmation: client.admin
Confirm only after refreshing every consumer, including disconnected ones and external key copies:
  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers

One of the remaining steps though I'm receiving the following:

Code: 
root@pve01:/etc/pve/priv/ceph# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
FAIL: error with 'auth commit-pending': mon_cmd failed - creating key with insecure key type ("`preferred' AKA aes") not allowed

Any suggestions? Ceph's overall status shows:

Code: 
root@pve03:/# ceph -s
  cluster:
    id:     354d9c2b-bdbf-49b5-aa66-46c841b39206
    health: HEALTH_WARN
            1 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
 
  services:
    mon: 3 daemons, quorum pve01,pve02,pve03 (age 15m) [leader: pve01]
    mgr: pve01(active, since 16m), standbys: pve02, pve03
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 3 up (since 11m), 3 in (since 11w)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 169 pgs
    objects: 51.22k objects, 199 GiB
    usage:   587 GiB used, 844 GiB / 1.4 TiB avail
    pgs:     169 active+clean
 
  io:
    client:   2.7 KiB/s rd, 13 KiB/s wr, 0 op/s rd, 4 op/s wr
 
So do yo know if I can run this step without affecting any live VMs ? My understanding is that I can run it and then I have to live migrate them no and run the same with --confirm-all-clients-refreshed ?

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
--rotate-all-storage-keys --rotate-admin-key
 
palachu said:
So do yo know if I can run this step without affecting any live VMs ? My understanding is that I can run it and then I have to live migrate them no and run the same with --confirm-all-clients-refreshed ?

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
--rotate-all-storage-keys --rotate-admin-key
Click to expand...
The above command worked once I applied the perl change at the top of my post. Had no issues with live VMs, just had to migrate them.

Where I'm hung, even now is with the command:

Code: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers

I've attempted this step now a couple of times, all of which fail. I've live migrated VMs and containers, rebooting the hypervisors one at a time, the above command failed. I've attempted shutting down all VMs and containers on the cluster and rebooting each hypervisor. Once again the command fails. Waiting a couple of days and attempting the run again receives:

Code: 
root@pve01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
FAIL: error with 'auth commit-pending': mon_cmd failed - creating key with insecure key type ("`preferred' AKA aes") not allowed

This is a test cluster, but I need to be able to get this to a successful completion so that I know what to do on my production clusters. Can someone from the Proxmox team provide any insight or assistance? I'm happy to supply any logs or command outputs to address this issue.
 
Thanks @RMaxwell72, Yes I am in the process of moving VMs and rebooting the nodes; I had some problems with nodes that ceph was still on version 19.2.3 rather than 19.2.6 since they were only clients and have no Manager, Monitor and OSD but those are fixed now; I still have like 400 sessions using the old ciphers (originally were more than 850) so tomorrow I will try to run the step 3.

Do you know if anyone has completed this successfully ?

Thank you.

PS: I have another 5 clusters to do after this.
 
HI @RMaxwell72,

So this worked for me. I executed the command twice. The first time it got rid of the admin key but fail querying the monitors.

Bash: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply \
    --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
PASS: 'client.admin' now uses the 'aes256k' cipher

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.
FAIL: refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified. Run this without options to check the remaining steps.

Then I run it a second time without trying to confirm the clients and it completed.

Bash: 
 /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphers
INFO: Collecting cluster info.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.

Apply this run, including recovery and any requested key confirmations? (y/N) y

Restricting the allowed ciphers
PASS: only the 'aes256k' cipher is allowed for authentication now; a key or client on the old cipher is refused from here on

What is left
No authentication check is an error: the service keys and the tickets they hand out are migrated.

PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
WARN: Keep /etc/pve/priv/cephx-key-migration.json until access verification. It contains secret keys needed to recover a daemon left behind ('ceph auth import'), so protect it like a keyring.

Hope it helps
 
No love:

Code: 
root@pve01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
FAIL: error with 'auth commit-pending': mon_cmd failed - creating key with insecure key type ("`preferred' AKA aes") not allowed
root@pve01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
FAIL: Cannot restrict the allowed ciphers to 'aes256k' yet:
  'client.admin': consumer refresh awaits your confirmation with '--confirm-clients-refreshed
    client.admin --apply'. Confirm only after refreshing every consumer, including disconnected
    ones and external key copies.

With this cluster, it is pointing to an auth commit-pending. So anyone have any ideas on how to get it from pending to committed?
 
Ok, so I found that running the following command:

Code: 
ceph mon set auth_preferred_cipher aes256k

and then running:

Code: 
root@pve01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
PASS: 'client.admin' now uses the 'aes256k' cipher
INFO: Nothing in this plan is stopped, so the cluster health does not gate this run.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.
WARN: 'auth_preferred_cipher' is already 'aes256k', so client keys created from now on will not work with kernel clients that do not know that cipher

Restricting the allowed ciphers
PASS: only the 'aes256k' cipher is allowed for authentication now; a key or client on the old cipher is refused from here on

What is left
No authentication check is an error: the service keys and the tickets they hand out are migrated.

Ceph still reports these authentication health checks:
  AUTH_INSECURE_CLIENT_KEY_TYPE: 1 auth client entities with insecure key types (current client keys use 'aes256k', not recomputed yet)
  AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types (cleared by this run, not recomputed yet)
These recompute on the monitors' next tick.
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
WARN: Keep /etc/pve/priv/cephx-key-migration.json until access verification. It contains secret keys needed to recover a daemon left behind ('ceph auth import'), so protect it like a keyring.

Done

This now cleared the warnings remaining in the WebUI. I then ran:

Code: 
root@pve01:~# ceph mon dump | grep -E 'auth_.*cipher'
dumped monmap epoch 27
auth_service_cipher aes256k
auth_allowed_ciphers aes256k
auth_preferred_cipher aes256k

The cluster is healthy at this time, and no issues remain.
 
You must log in or register to reply here.
Top Bottom
Back