but thats the grep command finding its own grep process in ps output, so pvestatd is indeed not running? Could you check on all three nodes
systemctl status pvestatd --no-pager -ljournalctl -u pvestatd -b -n 100 --no-pager or so?systemctl start pvestatd).And this is the first time the problem of the client key rotation in applications has been explained in a precise and clean manner. Two thumb up for it.
FYI: If you got a valid subscription that allows you accessing
pve-enterprise, then you can also use the ceph enterprise repository without having to do anything extra, just add it for all nodes (e.g. in the UI under Node -> Repositories -> Add) and disable the squid no-subscription one.Oh, I didn't know that <o>
Yes, then it's in sync with the PVE enterprise one, as we will coordinate the rollout of updates between our PVE and ceph enterprise repos.
That would be safe to do now and would make sense to ensure it's not overlooked for future updates.
root@proxmox01-pkl:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
FAIL: Could not verify 'aes256k' support for every service daemon:
could not verify the installed Ceph version of mon. on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of mon. on node 'proxmox02-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of mon. on node 'proxmox03-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of mgr.proxmox01-pkl on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of mgr.proxmox02-pkl on node 'proxmox02-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of osd.0 on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of osd.1 on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of osd.2 on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of osd.3 on node 'proxmox01-pkl.<domain>'; check that 'pvestatd' runs there
could not verify the installed Ceph version of osd.4 on node 'proxmox01-pkl.<domain>''; check that 'pvestatd' runs there
and 35 moreroot@proxmox01-pkl:~# ps aux | grep pvestatd
root 1219566 1.0 0.0 193616 126872 ? Ss 08:37 0:00 pvestatd
root 1219769 0.0 0.0 6528 2488 pts/2 S+ 08:38 0:00 grep pvestatdPVE 9.2.18 / pve-cluster 9.1.6
Ceph 19.2.6-pve4
PVE::Ceph::Services::get_ceph_versions() returns short keys:
proxmox01-pkl ... proxmox05-pkl
PVE::INotify::nodename() returns proxmox01-pkl
ceph mon metadata reports:
proxmox01-pkl.<domain>
The migration helper then reports installed Ceph versions as unknown.ceph health mute AUTH_INSECURE_SERVICE_KEY_TYPE
ceph health mute AUTH_INSECURE_SERVICE_TICKETS
ceph health mute AUTH_INSECURE_CLIENT_KEY_TYPE
ceph health mute AUTH_INSECURE_KEYS_ALLOWED
ceph health mute AUTH_INSECURE_KEYS_CREATABLE
ceph health mute AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE(MUTED) [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 2 auth client entities with insecure key types
entity client.admin using insecure key type: aes
entity client.pveuser using insecure key type: aes
(MUTED) [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types
insecure cipher aes allowed for auth
(MUTED) [WRN] AUTH_INSECURE_KEYS_CREATABLE: Monitors are configured to allow creation of insecure key types cluster:
id: ********-****-****-****-************
health: HEALTH_OK
(muted: AUTH_INSECURE_CLIENT_KEY_TYPE AUTH_INSECURE_KEYS_ALLOWED AUTH_INSECURE_KEYS_CREATABLE)
Sep 10 11:44:31 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:44:32 pve-1 pvestatd[1495]: mount error: Job failed. See "journalctl -xe" for details.
Sep 10 11:44:41 pve-1 pvestatd[1495]: mount error: Job failed. See "journalctl -xe" for details.
Sep 10 11:44:42 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:44:51 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:44:51 pve-1 pvestatd[1495]: mount error: Job failed. See "journalctl -xe" for details.
Sep 10 11:45:01 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:45:11 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:45:21 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:45:31 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:45:41 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:45:51 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:01 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:11 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:21 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:31 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:41 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:46:51 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:01 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:11 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:21 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:31 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:41 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:47:51 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:48:01 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:48:11 pve-1 pvestatd[1495]: rados_connect failed - Input/output erroSep 10 11:48:51 pve-1 pvestatd[1495]: rados_connect failed - Input/output error
Sep 10 11:48:54 pve-1 systemd[1]: Reloading pvestatd.service - PVE Status Daemon...
Sep 10 11:48:54 pve-1 pvestatd[53614]: send HUP to 1495
Sep 10 11:48:54 pve-1 pvestatd[1495]: received signal HUP
Sep 10 11:48:54 pve-1 pvestatd[1495]: server shutdown (restart)
Sep 10 11:48:54 pve-1 systemd[1]: Reloaded pvestatd.service - PVE Status Daemon.
Sep 10 11:48:55 pve-1 pvestatd[1495]: restarting server
on a hunch - are the steps here:
Node 1
root@pixi0:~# journalctl -u pvestatd -b --no-pager
Sep 09 11:50:29 pixi0 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: Unable to load access control list: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:50:30 pixi0 pvestatd[2970]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:50:30 pixi0 systemd[1]: pvestatd.service: Control process exited, code=exited, status=111/n/a
Sep 09 11:50:30 pixi0 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Sep 09 11:50:30 pixi0 systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.
Sep 09 11:50:30 pixi0 systemd[1]: pvestatd.service: Consumed 561ms CPU time, 108.5M memory peak.root@pixi1:~# journalctl -u pvestatd -b --no-pager
Sep 09 11:57:18 pixi1 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: Unable to load access control list: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:57:18 pixi1 pvestatd[2976]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:57:18 pixi1 systemd[1]: pvestatd.service: Control process exited, code=exited, status=111/n/a
Sep 09 11:57:18 pixi1 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Sep 09 11:57:18 pixi1 systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.
Sep 09 11:57:18 pixi1 systemd[1]: pvestatd.service: Consumed 563ms CPU time, 109.1M memory peak.root@pixi2:~# journalctl -u pvestatd -b --no-pager
Sep 09 11:59:59 pixi2 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: Unable to load access control list: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[1] failed: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[2] failed: Connection refused
Sep 09 11:59:59 pixi2 pvestatd[2956]: ipcc_send_rec[3] failed: Connection refused
Sep 09 11:59:59 pixi2 systemd[1]: pvestatd.service: Control process exited, code=exited, status=111/n/a
Sep 09 11:59:59 pixi2 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Sep 09 11:59:59 pixi2 systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.
Sep 09 11:59:59 pixi2 systemd[1]: pvestatd.service: Consumed 557ms CPU time, 109.1M memory peak.pve-cephx-rotate-service-keys runs as well.
In our environment, this issue is related to using FQDNs instead of short hostnames. By default, we use FQDNs everywhere. Example:
$ hostname
node1.our-domain.tld
$ hostname -f
node1.our-domain.tld # every daemon is judged on its Ceph version, only the touched ones on their data directory
for my $type (qw(mon mgr mds osd)) {
- $_->{binary} = $installed->{ $_->{node} } for $info->{daemons}->{$type}->@*;
+ for my $d ($info->{daemons}->{$type}->@*) {
+ my $short = (split(/\./, $d->{node} // ''))[0];
+ $d->{binary} = $installed->{$short} // $installed->{ $d->{node} };
+ }
}XXX@pvirtocbhpewd01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key --apply
INFO: Recovery and confirmations may run before the remaining plan can be shown.
Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: the 'cephfs' mount on node 'XXX01' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX02' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX03' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX04' is in use, left alone: /mnt/pve/cephfs: target is busy. It remains queued and keeps its previous key until an ordinary unmount succeeds.
Not touched by this run:
Additional actions not selected: cipher restriction.
Plan
Step 1: rotate 1 selected Ceph user key and rewrite every copy Proxmox VE keeps. Affected CephFS mounts are redone on every node unless something is using them.
Ceph user 'client.admin': staged next to the current key
For each staged Ceph user key, both the current and new keys authenticate until the new key is committed with '--confirm-clients-refreshed USER' or, once every open record is ready, '--confirm-all-clients-refreshed'.
Live sessions of the selected Ceph users:
'client.admin', 9 sessions: XXX01: 2, XXX02: 3, XXX03: 1, XXX04: 3
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
XXX01: possible kernel client
XXX02: possible kernel client
XXX03: possible kernel client
XXX04: possible kernel client
For staged keys, refresh their consumers after applying: live-migrate every VM, remount every kernel CephFS or RBD mount, and restart other consumers; a dry run then reports who is left.
INFO: the new key of 'client.admin' was staged by an earlier run, rewriting its copies
PASS: a fresh 'client.admin' connection succeeds with the staged key
WARN: the 'cephfs' mount on node 'XXX01' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX02' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX03' is timed out, left alone: the unmount failed. It remains queued and keeps its previous key until an ordinary unmount succeeds.
WARN: the 'cephfs' mount on node 'XXX04' is in use, left alone: /mnt/pve/cephfs: target is busy. It remains queued and keeps its previous key until an ordinary unmount succeeds.
PASS: 'client.admin' has a new 'aes256k' key staged next to its current one; both authenticate until you commit the new one, once every consumer of it is refreshed.
What is left
No authentication check is an error: the service keys and the tickets they hand out are migrated.
Ceph still reports these authentication health checks:
AUTH_INSECURE_CLIENT_KEY_TYPE: 2 auth client entities with insecure key types
AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types
AUTH_INSECURE_KEYS_CREATABLE: Monitors are configured to allow creation of insecure key types
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types
These recompute on the monitors' next tick; AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE clears on its own within a few hours.
'ceph health mute <check>' silences the warning about a client key you leave on the old cipher.
Client keys awaiting action:
'client.admin': the new key is staged. Both keys remain valid. CephFS refresh pending: 'cephfs'
on node 'XXX01', 'cephfs' on node 'XXX02', 'cephfs' on node
'XXX03', 'cephfs' on node 'XXX04'. Free busy mounts and resolve node or
mount errors, then rerun with '--apply'.
Left to whoever manages the client that reads them; 'man pveceph' covers what each needs:
client.k8s
WARN: Keep /etc/pve/priv/cephx-key-migration.json until migration completion and access verification. It contains secret keys needed to recover a daemon left behind ('ceph auth import'), so protect it like a keyring.
DoneYep exactly the same thing for us.
ceph auth rotate --key-type=aes256k client.RIPsmigrations/pve-cephx-rotate-service-keysoot@proxh2:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.
These keys need manual migration. Follow each client's key-rotation procedure; update every key copy and refresh every client:
client.filesHalle
client.mirror
client.mirror_remoteThese don't look like they were created by Proxmox so they need to be rotated manually. One question, obviously, is whether these clients can actually consume the aes256k key to begin with. You may need to first update the mirror daemons or whatever's on the other end. Userspace clients need Ceph 19.2.6 or 20.2.4, and kernel mounts need kernel 7.0 or a distribution backport.