It looks more complicated than it is. Run the first command and the script output will tell you the next one to run. Until the last one you can just stop/pause.
the -pve suffix indicates patches by Proxmox. If you look at the updates tab in the GUI (as root) you can see pending updates. If you don’t already see the keys warning in Ceph in the web GUI you’re not on the latest.
not sure if we have different script versions but only the final 'confirm' step gave me the next command to run. up until that point during 'dryrun', if anything remained undone on that step, the output would tell you to 'review plan and run this again with --apply'...but this is awkward. in my opinion, it feels wrong or 'too soon' to rerun a potentially dangerous ceph altering script with the '--apply' before handling the 'remaining things to do' and then running the DRYRUN again as a test to see if your efforts were successful and to see if anything is left on the 'remaining things to do' section. if i was writing the helper script, i wouldnt be suggesting that users run the script again with --apply after only running the dryrun once, and with potentially a bunch of things left to fix on that step. it just felt awkwardly worded and weird pacing and weird instructions. also the on the second step where you possibly need to migrate vms off the node to clear their keys, after doing your first '--apply' run and you are left with some 'remaining' things, the script output very clearly does not tell you what to run next. here is mine verbatim from a moment ago, for example: (my 5th cluster doing this upgrade since yesterday :X )
Code:
What is left
No authentication check is an error: the service keys and the tickets they hand out are migrated.
Ceph still reports these authentication health checks:
AUTH_INSECURE_CLIENT_KEY_TYPE: 1 auth client entities with insecure key types
AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types
AUTH_INSECURE_KEYS_CREATABLE: Monitors are configured to allow creation of insecure key types
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types
These recompute on the monitors' next tick; AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE clears on its own within a few hours.
Client keys awaiting action:
'client.admin': 7 session(s) still authenticate with a previous key (c1-pve01: 3,
c1-pve02: 2, c1-pve03: 1, c1-pve04: 1). Both keys remain valid. Refresh these
consumers, then rerun without options.
Possible consumers (host-wide hints, not session attribution):
c1-pve01: 3 unattributed sockets without an owning process (possible kernel clients)
c1-pve02: 3 unattributed sockets without an owning process (possible kernel clients)
c1-pve03: 2 unattributed sockets without an owning process (possible kernel clients)
c1-pve04: 2 unattributed sockets without an owning process (possible kernel clients)
Session absence does not prove that disconnected consumers or external key copies were refreshed.
WARN: Keep /etc/pve/priv/cephx-key-migration.json until migration completion and access verification. It contains secret keys needed to recover a daemon left behind ('ceph auth import'), so protect it like a keyring.
Done
the output says 'what is left' but then 'no authentication check is an error: the service keys and the tickets they hand out are migrated.'. no disrespect to any english as a second language speakers or the devs (i love you guys for all you do) that wrote this helper script, but this is a really oddly worded and confusing for me as an english speaker. i can sort of try to feel it out and guess what it is referring to, but in something this critical to ceph health/client uptime/data loss/etc, im ultra careful and hesitant to make any sudden moves just to be safe. add the not-as-helpful-as-it-could-be script in with that and it's not the best time for me or im sure many others. that's why i came to the forums yesterday to begin with, i wanted to double-check and see if anyone posted slightly more clear instructions or to try get a better idea about whats going on with this whole upgrade since i was worried about moving forward after running a dry run a few times. also 'possible consumers' is very confusing as well. ive never heard of any part of ceph being referred to as a 'consumer' before so it's hard for me to understand what exactly it's trying to tell me. the word 'consumer' only appears twice in the enitre documentation for deploying hyper converged ceph documentation, and one of those times is in reference to 'consumer hardware'. in the documentation there is a mention of migrating vms off nodes, and the mention of 'sockets/kernel clients' i pieced the two together and guessed (not something you want to be doing in important situations like this) that it meant that is a list of nodes with # of vms to be migrated off to clear keys. ended up being right, but again, hoping for some more clarity or reworked instructions/script to clear some of these confusing parts up. anyway, you can see after one 'dryrun' it just gives you that list (hope you can figure out what it means), then just says 'done.' not even suggesting the 'run again with apply' this time. ¯\_(ツ)_/¯ itd be nice for some guidance and suggest that after trying to knock out some things on the list, for the user to run the 'dryrun' again to see what worked/what didnt/whats left. then after multiple fix/dryruns, finally the dryrun comes back with something new about 'all stuff is done...now run this command next' or something along those lines. the third step more or less does that, but all in all, this whole thing could've been better handled, both the surprise shock of it all of a sudden appearing once you met the kernel/ceph version criteria (it felt like it was hibernating and sprung up like a trap once i did the very minor update that happened to get me across the kernel requirement line)
for those curious, this is what that final output looks like after all of your stuff in step 2 is done. the script will pop out this message, and that should be your sign that you are good to go forward:
Code:
PASS: Every service key uses 'aes256k', and so do the service tickets.
Ready for confirmation: client.admin
Confirm only after refreshing every consumer, including disconnected ones and external key copies:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers
anyway, all love and respect to all the PVE and ceph devs out there, you are all great and i very much appreicate all you do to make this all possible and our business possible. hope maybe this can be seen as some constructive criticism and a friendly heads up that this whole key migration has been a bit of a misfire (at least as far as an everyday admin is concerned), and i hope maybe someone there can spend a little time rewording and improving the documentation/helper script a bit to help make things a little more understandable and clear for anyone who runs into this upgrade over the next few days/weeks/whatever. thanks again.
quick edit: oh in case anyone missed it...run all your commands/scripts with the '--verbose' flag as well. it adds a little bit more info/clarity (but dont expect too much). still better than nothing. ok good luck!
