[SOLVED] PVE-Cluster fails to start after time change

poisonborz

Member
May 8, 2020
25
2
23
54
Using the latest proxmox, I changed the system time (to earlier) - I assume this is the problem, as this is the only change I did that could result in this - and after a restart PVE-Cluster fails to start.

Code:
systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
systemd[1]: pve-cluster.service: Start request repeated too quickly.
systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

The PVE-proxy has a bit more detailed log:

Code:
systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
pvecm[1235]: ipcc_send_rec[1] failed: Connection refused
pvecm[1235]: ipcc_send_rec[2] failed: Connection refused
pvecm[1235]: ipcc_send_rec[3] failed: Connection refused
pvecm[1235]: Unable to load access control list: Connection refused
pveproxy[1237]: starting server
pveproxy[1237]: starting 3 worker(s)
pveproxy[1237]: worker 1238 started
pveproxy[1237]: worker 1239 started
pveproxy[1237]: worker 1240 started
pveproxy[1238]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.


How could this be fixed?
 
Hello

Yes, Errors like this can be expected after changing the time. You get this situations where things try to authenticate with credentials from the future.

Does pvecm updatecerts --force fix your problem?
 
Sadly doesn't seems so:

Code:
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
 
Sadly doesn't seems so:

Code:
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
did you run this a privileged user? (root)

If so please attach the output of (as root) journalctl -b for better diagnostic
 
Last edited:
Yes. The only unique lines in journalctl -b are:

Code:
localhost pveproxy[1644]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
localhost cron[1120]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
 
can you see if ls -l /etc/pve works?

also what does systemctl status pve-cluster say?
 
Last edited:
/etc/pve is empty, probably because the node can't connect (as per forum searches for the same problem).

I've posted parts of status pve-cluster above, the full response is

Code:
pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-09-04 12:10:49 CEST; 2h 34min ago
    Process: 1231 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 8ms

localhost systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
localhost systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
localhost systemd[1]: pve-cluster.service: Start request repeated too quickly.
localhost systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
localhost systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
 
The actual errors for the failing start of pve-cluster are not visible, please check the full unit log from the current boot for errors at the start:
journalctl -b -u pve-cluster.service
 
Thanks for the response, but I simply performed a full reinstall since, so I can't research this anymore. I marked the thread as "solved" still, although it would be useful to know how to remedy this kind of situation.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!