[SOLVED] PVE-Cluster fails to start after time change

[poisonborz]

Using the latest proxmox, I changed the system time (to earlier) - I assume this is the problem, as this is the only change I did that could result in this - and after a restart PVE-Cluster fails to start.

systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
systemd[1]: pve-cluster.service: Start request repeated too quickly.
systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

The PVE-proxy has a bit more detailed log:

systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
pvecm[1235]: ipcc_send_rec[1] failed: Connection refused
pvecm[1235]: ipcc_send_rec[2] failed: Connection refused
pvecm[1235]: ipcc_send_rec[3] failed: Connection refused
pvecm[1235]: Unable to load access control list: Connection refused
pveproxy[1237]: starting server
pveproxy[1237]: starting 3 worker(s)
pveproxy[1237]: worker 1238 started
pveproxy[1237]: worker 1239 started
pveproxy[1237]: worker 1240 started
pveproxy[1238]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.

How could this be fixed?

 

[Philipp Hufnagl]

Hello

Yes, Errors like this can be expected after changing the time. You get this situations where things try to authenticate with credentials from the future.

Does pvecm updatecerts --force fix your problem?

 

[poisonborz]

Sadly doesn't seems so:

ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

 

[Philipp Hufnagl]

Sadly doesn't seems so:

ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

did you run this a privileged user? (root)

If so please attach the output of (as root) journalctl -b for better diagnostic

 

[poisonborz]

Yes. The only unique lines in journalctl -b are:

localhost pveproxy[1644]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
localhost cron[1120]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)

 

[Philipp Hufnagl]

can you see if ls -l /etc/pve works?

also what does systemctl status pve-cluster say?

 

[poisonborz]

/etc/pve is empty, probably because the node can't connect (as per forum searches for the same problem).

I've posted parts of status pve-cluster above, the full response is

pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-09-04 12:10:49 CEST; 2h 34min ago
    Process: 1231 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 8ms

localhost systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
localhost systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
localhost systemd[1]: pve-cluster.service: Start request repeated too quickly.
localhost systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
localhost systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

 

[t.lamprecht]

The actual errors for the failing start of pve-cluster are not visible, please check the full unit log from the current boot for errors at the start:
journalctl -b -u pve-cluster.service

 

[poisonborz]

Thanks for the response, but I simply performed a full reinstall since, so I can't research this anymore. I marked the thread as "solved" still, although it would be useful to know how to remedy this kind of situation.