Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon

jsterr said:
You do not need to reboot the nodes, and NEVER use apt upgrade -> it can break your system. use dist-upgrade or full-upgrade
Click to expand...
OK. Thank you very much for your input. Understood to never use simple apt upgrade

But the kernel would be also updated, so I think we have to restart the nodes on some point?

If I understood you correctly:
1) Install ALL updates directly on all three nodes with apt full-upgrade
2) Do the rolling restart of Ceph services
3) Do the key exchange
4) Let go all the errors and warnings regarding CephX authentication
5) Then I would restart the nodes? (because of the kernel updates from step 1)

Honestly, we're running PVE since years (12-13?) now and as I can remember we never bricked our systems with the simple apt upgrade but if you had bad experiences in the past then we should avoid that in the future. Thank you
:)
 
shfp79 said:
OK. Thank you very much for your input. Understood to never use simple apt upgrade

But the kernel would be also updated, so I think we have to restart the nodes on some point?

If I understood you correctly:
1) Install ALL updates directly on all three nodes with apt full-upgrade
2) Do the rolling restart of Ceph services
3) Do the key exchange
4) Let go all the errors and warnings regarding CephX authentication
5) Then I would restart the nodes? (because of the kernel updates from step 1)

Honestly, we're running PVE since years (12-13?) now and as I can remember we never bricked our systems with the simple apt upgrade but if you had bad experiences in the past then we should avoid that in the future. Thank you
:)
Click to expand...

Theoretically apt upgrade might refuse to install, remove, or replace package dependencies that Proxmox requires to function. Yes you can reboot at a different time. You also might need to move around some vms between hosts to "refresh" clients as the docs state.
 
I don't remember the commands but following the upstream docs worked for me. Remember to make sure the client is recent enough for the new key.
 
  • Like
Reactions: shfp79
Thank you very much. So we followed upstream docs to rotate the external CephFS client:
Code: 
ceph auth rotate --key-type=aes256k client.client_name | tee ./client.client_name.keyring

Then ran again the helper script which led to:
Code: 
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
WARN: the key of 'client.client_name' changed outside this script; retaining recorded instances and observed sessions not identified with the target key
PASS: Every service key uses 'aes256k', and so do the service tickets.

Client keys awaiting action:
  'client.client_name': the first complete client measurement is pending. After refreshing every
    client, run '--confirm-clients-refreshed client.client_name --apply'; the first attempt records the
    measurement without committing the key.

So we ran /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --confirm-clients-refreshed client.client_name --apply

which led to:
Code: 
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: the key of 'client.client_name' changed outside this script; retaining recorded instances and observed sessions not identified with the target key
FAIL: not accepting '--confirm-clients-refreshed client.client_name' yet: its rotation needed a first complete measurement; no possibly stale session is currently visible, and that complete measurement is now recorded. Repeat the confirmation after refreshing any recorded clients; it then covers only clients this run cannot see.

Another dry-run with
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys led to the following output:

Code: 
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Ready for confirmation: client.client_name
Confirm only after refreshing every client, including disconnected ones and external key copies:
  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers

OK, here we go:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers

Output:
Code: 
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.client_name: every client of them that this run can see is refreshed, and you confirm the same for those it cannot see

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.

Restricting the allowed ciphers
PASS: only the 'aes256k' cipher is allowed for authentication now; a key or client on the old cipher is refused from here on

What is left
No authentication check is an error: the service keys and the tickets they hand out are migrated.

Ceph still reports these authentication health checks:
  AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types (cleared by this run, not recomputed yet)
  AUTH_INSECURE_KEYS_CREATABLE: Monitors are configured to allow creation of insecure key types (cleared by this run, not recomputed yet)
These recompute on the monitors' next tick.
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
WARN: Keep /etc/pve/priv/cephx-key-migration.json until access verification. It contains secret keys needed to recover a daemon left behind ('ceph auth import'), so protect it like a keyring.

Done


I think, the information to rotate external clients as in our example could also be provided by the official docs or in the Proxmox Wiki.
 
You must log in or register to reply here.
Top Bottom
Back