[SOLVED] pve-cephx-rotate-service-keys error in restrict-ciphers phase "not every monitor answered the session query"

[zukka]

Hi, I'm following the instructions to make the aes -> aes256k ceph key migration.
I've a 3 node cluster, with all the updates installed.
Everything went fine until I had to run the last step:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Remaining migration steps:
  --restrict-ciphers: allow only the 'aes256k' cipher for authentication, once every running consumer holds its rotated key
Next command:
  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphers

I get this error:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --restrict-ciphers --apply
INFO: Collecting cluster info.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.

Apply this run, including recovery and any requested key confirmations? (y/N) y
FAIL: refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified. Run this without options to check the remaining steps.

While doing the procedure i briefly see some mon server going down in the ceph web interface.

How can I debug what's going under the hood and test these session query ?

thank you

 

[zukka]

Launching the script on node 3 did the trick.
I'm still curious to know why didn't work from node 1 and 2.

 

[alext]

Thanks for your update! Same thing here, only from one of the nodes did the command take (even though the dry-run said everything is fine).
Running on a 5-node cluster.

 

[wursti]

That Trick didnt work for me.
Tried on a 4-Node Cluster (PVE 9.2.20 / Linux 7.0.14-17-pve)

 

[gosha2818]

I ended up having to do a rolling monitor restart and then ran from the node of the last monitor restarted.

 

[jalet]

Adding a data point since this seems to be affecting more than isolated setups.

Environment: 5-monitor Proxmox+Ceph 6-nodes cluster stretched across two datacenters, with one witness-only monitor at a third site. Ceph 20.2.4, kernel 7.0.14-19-pve on all nodes. All service/client keys already showed aes256k with 0 pending keys, and dry run reported clean (PASS, no issues).

Ran `--apply --restrict-ciphers --verbose` and hit the same error as above: "refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified."

Before trying anything else, verified that the mon keyring and client.admin keyring were identical across every monitor host, including the witness (compared /etc/pve/priv/ceph.mon.keyring, /var/lib/ceph/mon/ceph-<witness>/keyring, and both client.admin keyring locations) — no mismatch there.

Retried the exact same apply command from each monitor host in turn. It failed from the 1st through 4th hosts, then succeeded cleanly from the 5th. No mon restarts, no other changes in between — just re-running from a different host each time.

So like the other reports here, it's inconsistent and host-dependent with no obvious pattern, but this may be the first report from a stretched cluster with a witness monitor in the mix rather than a standard single-site cluster — in case that's a relevant variable for whoever ends up tracking this down.

I'll do the very same Proxmox+Ceph upgrade on our second 8-nodes cluster (also stretched with tie-breaker on 3rd site) and see how it goes.

 

[jalet]

No problem on the second cluster, although I had to ensure that the tie-breaker's ssh host key was first added to ~root/.ssh/known_hosts on the node I launched the /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys command on, otherwise it failed without telling exactly why.