icc_send_rec[1] failed: Connection refused

[davidand]

One of the two nodes of my Proxmox cluster won't start after reboot.

This is what I tried:

$ pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

$ tail -100 /var/log/syslog
...
Oct 10 17:29:08 proxmox-2 pveproxy[912]: worker 1243 finished
Oct 10 17:29:08 proxmox-2 pveproxy[912]: starting 2 worker(s)
Oct 10 17:29:08 proxmox-2 pveproxy[912]: worker 1249 started
Oct 10 17:29:08 proxmox-2 pveproxy[912]: worker 1250 started
Oct 10 17:29:08 proxmox-2 pveproxy[1249]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1891.
Oct 10 17:29:08 proxmox-2 pveproxy[1250]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1891.

$ systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-10-10 17:23:46 CEST; 5min ago
    Process: 910 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 911 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)

$ /usr/bin/pvecm updatecerts
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

$ cat /etc/hostname
proxmox-2

$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.88.6 proxmox-2.local proxmox-2

# The following lines are desirable for IPv6 capable hosts
.....

So I tried to remove this node from the cluster using this guide here: https://forum.proxmox.com/threads/proxmox-ve-6-removing-cluster-configuration.56259/

It did not help I'm having big troubles now because the VM on this node is dead and I need it to be running (((

$ systemctl stop pve-cluster

$ systemctl stop corosync

$ pmxcfs -l
[main] notice: forcing local mode (although corosync.conf exists)
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
[main] crit: fuse_mount error: File exists
[main] notice: exit proxmox configuration filesystem (-1)

$ rm /etc/pve/corosync.conf
rm: cannot remove '/etc/pve/corosync.conf': No such file or directory

$ rm -r /etc/corosync/*
rm: cannot remove '/etc/corosync/*': No such file or directory

$ killall pmxcfs
pmxcfs: no process found
root@proxmox-2:~# systemctl start pve-cluster
Job for pve-cluster.service failed because the control process exited with error code.
See "systemctl status pve-cluster.service" and "journalctl -xe" for details.

$ pvecm delnode proxmox-2
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused

 

[davidand]

Ok, I managed to resolve my problem thanks to this post: https://forum.proxmox.com/threads/unable-to-load-access-control-list-connection-refused.72245/page-2

TL;DR

I had to remove the /etc/pve folder on my second node (as storage.cfs config got stuck in there and paralyzed the entire system)

In case it helps anybody