[SOLVED] pve-cephx-rotate-service-keys error in restrict-ciphers phase "not every monitor answered the session query"

[zukka]

Hi, I'm following the instructions to make the aes -> aes256k ceph key migration.
I've a 3 node cluster, with all the updates installed.
Everything went fine until I had to run the last step:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Remaining migration steps:
  --restrict-ciphers: allow only the 'aes256k' cipher for authentication, once every running consumer holds its rotated key
Next command:
  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphers

I get this error:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --restrict-ciphers --apply
INFO: Collecting cluster info.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.

Apply this run, including recovery and any requested key confirmations? (y/N) y
FAIL: refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified. Run this without options to check the remaining steps.

While doing the procedure i briefly see some mon server going down in the ceph web interface.

How can I debug what's going under the hood and test these session query ?

thank you

 

[zukka]

Launching the script on node 3 did the trick.
I'm still curious to know why didn't work from node 1 and 2.

 

[alext]

Thanks for your update! Same thing here, only from one of the nodes did the command take (even though the dry-run said everything is fine).
Running on a 5-node cluster.