[SOLVED] pve-cluster.service doesn't start and ipcc_send_rec failed

[BrotherJ]

Hello, I have on

# pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-15-pve)

the problem that the VMs no longer start.
I had a full /var partition, in /lib/pve/local-btrfs/images there were incorrectly VM images that actually belonged in a ZFS pool, and I replaced this with a larger partition (LVM). I synced everything from the old /var to the new /var. And after rebooting the server I wanted to come along 'qm list' the VMs check but only get the following

# qm list
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

In the meantime I noticed that the pve-cluster.service was not started:

# systemctl status pve-cluster.service
× pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-05-01 11:50:17 CEST; 8min ago
    Process: 8392 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 10ms

May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
May 01 11:50:17 neckar1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Start request repeated too quickly.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
May 01 11:50:17 neckar1 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

and

# journalctl -xeu pve-cluster.service
May 01 11:50:16 neckar1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:16 neckar1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:16 neckar1 pmxcfs[8390]: fuse: mountpoint is not empty
May 01 11:50:16 neckar1 pmxcfs[8390]: fuse: if you are sure this is safe, use the 'nonempty' mount option
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] crit: fuse_mount error: File exists
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] crit: fuse_mount error: File exists
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:16 neckar1 pmxcfs[8390]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:16 neckar1 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
May 01 11:50:16 neckar1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
May 01 11:50:16 neckar1 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:16 neckar1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
May 01 11:50:16 neckar1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:16 neckar1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:16 neckar1 pmxcfs[8391]: fuse: mountpoint is not empty
May 01 11:50:16 neckar1 pmxcfs[8391]: fuse: if you are sure this is safe, use the 'nonempty' mount option
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] crit: fuse_mount error: File exists
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] crit: fuse_mount error: File exists
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:16 neckar1 pmxcfs[8391]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:16 neckar1 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
May 01 11:50:16 neckar1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
May 01 11:50:16 neckar1 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
May 01 11:50:17 neckar1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:17 neckar1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] notice: resolved node name 'neckar1' to '192.55.84.250' for default node IP address
May 01 11:50:17 neckar1 pmxcfs[8392]: fuse: mountpoint is not empty
May 01 11:50:17 neckar1 pmxcfs[8392]: fuse: if you are sure this is safe, use the 'nonempty' mount option
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] crit: fuse_mount error: File exists
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] crit: fuse_mount error: File exists
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:17 neckar1 pmxcfs[8392]: [main] notice: exit proxmox configuration filesystem (-1)
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
May 01 11:50:17 neckar1 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
May 01 11:50:17 neckar1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Start request repeated too quickly.
May 01 11:50:17 neckar1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
May 01 11:50:17 neckar1 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

and

pvecm --help
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

Do you have any ideas on how to start this again?

Best regards
BrotherJ

 

[BrotherJ]

I was able to fix this by deleting '/etc/pve' and restarting the pve-cluster.service. The fuse message in the log made me prick up my ears. Now I only have the old VM configs, which don't all start.