Cannot start KVM

eddi1984

Active Member
Oct 8, 2014
46
0
26
Hi,

I had a situation, where (it looks like, cannot quite pinpoint the problem) one VM (Win2011 SBS Std) will start using close to 80-90% of CPU and lockup the proxmox server (server load was 30 (max)). Everything becomes VERY slow and unresponsive. It was so bad, that I could not open a console (noVNC or SPICE) to investigate. No backup was running at that time.

In order for me to get the server back to working condition, I decided to just hard STOP the VM (not a clean shutdown). After that, everything settled down and seems to work fine now. Than I tried to START the VM again. No luck. I get this here:
Code:
Failed to create message: Input/output error
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 100 -p 'CPUShares=1000' /usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -name WinSBS2011Std -smp '4,sockets=2,cores=2,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga qxl -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,-kvm_steal_time,enforce' -m 32768 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -spice 'tls-port=61002,addr=localhost,tls-ciphers=DES-CBC3-SHA,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4827fbaece2b' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/storage/vms/vm-100-disk-2,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -drive 'file=/dev/zvol/storage/vms/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=101' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=XX:XX:XX:XX:XX:XX,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1

I tried to restart the pve-cluster and pvestatd. I also unlocked the VM. All the restart and unlock commands worked and did not return an error but the VM will still not start.

This is the output of pveversion -v:
Code:
root@VMNode01:/# pveversion -v
proxmox-ve: 4.0-22 (running kernel: 4.2.3-2-pve)
pve-manager: 4.0-57 (running version: 4.0-57/cc7c2b53)
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1
libqb0: 0.17.2-1
pve-cluster: 4.0-24
qemu-server: 4.0-35
pve-firmware: 1.1-7
libpve-common-perl: 4.0-36
libpve-access-control: 4.0-9
libpve-storage-perl: 4.0-29
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-12
pve-container: 1.0-21
pve-firewall: 2.0-13
pve-ha-manager: 1.0-13
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.4-3
lxcfs: 0.10-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie

I have not restarted the node, however, the server is in production and I would have to wait until I can restart it. I am hoping to resolve the issue without a restart.

If you guys need any other info, please let me know.

THANK YOU!!!
 
Last edited:
Update:

I had no option, but to get the node restarted. The VM started up fine after the reboot.

But now I am getting these errors in syslog:
Code:
Dec  9 12:08:58 VMNode01 pvedaemon[4738]: <root@pam> starting task UPID:VMNode01:00002B0D:0001C41D:56687C4A:srvrestart:pve-cluster:root@pam:
Dec  9 12:08:58 VMNode01 pvedaemon[11021]: re-starting service pve-cluster: UPID:VMNode01:00002B0D:0001C41D:56687C4A:srvrestart:pve-cluster:root@pam:
Dec  9 12:08:58 VMNode01 pmxcfs[9710]: [main] notice: teardown filesystem
Dec  9 12:08:59 VMNode01 pve-ha-lrm[10753]: unable to write lrm status file - unable to open file '/etc/pve/nodes/VMNode01/lrm_status.tmp.10753' - Software caused connection abort
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:08:59 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:00 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:01 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 cron[4494]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:02 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:09:03 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pve-ha-crm[4742]: status change slave => wait_for_quorum
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:03 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:09:04 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:04 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:05 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:06 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:07 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:08 VMNode01 systemd[1]: pve-cluster.service stop-sigterm timed out. Killing.
Dec  9 12:09:08 VMNode01 systemd[1]: pve-cluster.service: main process exited, code=killed, status=9/KILL
Dec  9 12:09:08 VMNode01 systemd[1]: Unit pve-cluster.service entered failed state.
Dec  9 12:09:09 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvecm[11147]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvecm[11147]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvecm[11147]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pveproxy[4746]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:09 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Connection refused
Dec  9 12:09:18 VMNode01 pve-ha-crm[4742]: status change wait_for_quorum => slave
Dec  9 12:09:45 VMNode01 pveproxy[10260]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:09:45 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:10:01 VMNode01 cron[4494]: (*system*vzdump) RELOAD (/etc/cron.d/vzdump)
Dec  9 12:10:15 VMNode01 pveproxy[4745]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:10:50 VMNode01 pvedaemon[4737]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:11:00 VMNode01 pvedaemon[12474]: starting vnc proxy UPID:VMNode01:000030BA:0001F394:56687CC4:vncproxy:100:root@pam:
Dec  9 12:11:00 VMNode01 pvedaemon[4737]: <root@pam> starting task UPID:VMNode01:000030BA:0001F394:56687CC4:vncproxy:100:root@pam:
Dec  9 12:11:01 VMNode01 cron[4494]: (*system*vzdump) RELOAD (/etc/cron.d/vzdump)
Dec  9 12:11:10 VMNode01 pve-ha-crm[4742]: successfully acquired lock 'ha_manager_lock'
Dec  9 12:11:10 VMNode01 pve-ha-crm[4742]: watchdog active
Dec  9 12:11:10 VMNode01 pve-ha-crm[4742]: status change slave => master

I now have the ipcc_send_rec error. I never had that before. After doing some reading, it looks like, it belongs to HA. However, I only have the one node, no cluster. The other thing is, that quorum shows 2 entries (see picture)???
HA is shown as enabled when checking the running services. Is this normal?

Thanks.
 
To fix that , try

# systemctl restart pve-cluster

But you should update your installation - that bug is already fixed with new packages from pve-no-subscription repository.
 
Hi Dietmar,

I updated from the pve-no-subscription repo approx 2 weeks ago. When was this fixed?

# systemctl restart pve-cluster executed fine, this is the log after I ran it:
Code:
root@VMNode01:~# tail -f /var/log/syslog
Dec  9 12:59:15 VMNode01 pvecm[24892]: ipcc_send_rec failed: Connection refused
Dec  9 12:59:16 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:59:20 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:59:41 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:59:47 VMNode01 pveproxy[18300]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:59:47 VMNode01 pvedaemon[4737]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 12:59:47 VMNode01 pvedaemon[4737]: <root@pam> successful auth for user 'root@pam'
Dec  9 12:59:47 VMNode01 pveproxy[17955]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:00:03 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:00:50 VMNode01 pvedaemon[4739]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:03:16 VMNode01 pmxcfs[24890]: [main] notice: teardown filesystem
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:03:17 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:17 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:18 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:18 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:18 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:19 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:20 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:03:20 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:20 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:20 VMNode01 pve-ha-crm[4742]: lost lock 'ha_manager_lock - can't create '/etc/pve/priv/lock' (pmxcfs not mounted?)
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:03:21 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:21 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:22 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:23 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:24 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: status change master => lost_manager_lock
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: watchdog closed (disabled)
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: status change lost_manager_lock => wait_for_quorum
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:25 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:26 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 systemd[1]: pve-cluster.service stop-sigterm timed out. Killing.
Dec  9 13:03:27 VMNode01 systemd[1]: pve-cluster.service: main process exited, code=killed, status=9/KILL
Dec  9 13:03:27 VMNode01 systemd[1]: Unit pve-cluster.service entered failed state.
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvecm[27003]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvecm[27003]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pvecm[27003]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:27 VMNode01 pveproxy[19143]: ipcc_send_rec failed: Connection refused
Dec  9 13:03:35 VMNode01 pve-ha-crm[4742]: status change wait_for_quorum => slave

It looks like its not helping. Bug?
 
Me again.

I added the pve-no-subscription repo to the list and ran an update. There was a long list and I did a dist-upgrade. Everything was installed fine, however, the problem persist.

Log of the upgrade and I also did a # systemctl restart pve-cluster:
Code:
Dec  9 13:18:08 VMNode01 pvedaemon[30217]: launch command: /usr/bin/vncterm -rfbport 5900 -timeout 10 -authpath /nodes/VMNode01 -perm Sys.Console -notls -listen localhost -c /bin/bash -c pveupgrade --shell
Dec  9 13:18:30 VMNode01 pveproxy[17955]: worker exit
Dec  9 13:18:30 VMNode01 pveproxy[4744]: worker 17955 finished
Dec  9 13:18:30 VMNode01 pveproxy[4744]: starting 1 worker(s)
Dec  9 13:18:30 VMNode01 pveproxy[4744]: worker 30270 started
Dec  9 13:19:41 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:19:41 VMNode01 fusermount[3447]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Dec  9 13:19:42 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:19:50 VMNode01 kernel: [ 5412.473055] vmbr0: port 5(veth201i0) entered disabled state
Dec  9 13:19:50 VMNode01 kernel: [ 5412.474526] device veth201i0 left promiscuous mode
Dec  9 13:19:50 VMNode01 kernel: [ 5412.474530] vmbr0: port 5(veth201i0) entered disabled state
Dec  9 13:21:03 VMNode01 pmxcfs[27001]: [main] notice: teardown filesystem
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: lost lock 'ha_manager_lock - can't create '/etc/pve/priv/lock' (pmxcfs not mounted?)
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 systemd[1]: pve-cluster.service stop-sigterm timed out. Killing.
Dec  9 13:21:14 VMNode01 systemd[1]: pve-cluster.service: main process exited, code=killed, status=9/KILL
Dec  9 13:21:14 VMNode01 systemd[1]: Unit pve-cluster.service entered failed state.
Dec  9 13:21:14 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:15 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:15 VMNode01 lxcfs[6088]: hierarchies: 1: name=systemd
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 2: cpuset
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 3: cpu,cpuacct
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 4: blkio
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 5: memory
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 6: devices
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 7: freezer
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 8: net_cls,net_prio
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 9: perf_event
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 10: hugetlb
Dec  9 13:21:15 VMNode01 kernel: [ 5497.556715] cgroup: new mount options do not match the existing superblock, will be ignored
Dec  9 13:21:16 VMNode01 kernel: [ 5497.827361] audit: type=1400 audit(1449692476.150:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/lxc-start" pid=6096 comm="apparmor_parser"
Dec  9 13:21:16 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:16 VMNode01 lxc-devsetup[6120]: /dev is devtmpfs
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: status change master => lost_manager_lock
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: watchdog closed (disabled)
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: status change lost_manager_lock => wait_for_quorum
Dec  9 13:21:22 VMNode01 pve-ha-crm[4742]: status change wait_for_quorum => slave
Dec  9 13:21:28 VMNode01 pvepw-logger[4163]: received terminate request (signal)
Dec  9 13:21:28 VMNode01 pvepw-logger[4163]: stopping pvefw logger
Dec  9 13:21:28 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:28 VMNode01 pvepw-logger[11250]: starting pvefw logger
Dec  9 13:21:28 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: received signal TERM
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: server closing
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: clear firewall rules
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: server stopped
Dec  9 13:21:30 VMNode01 pve-firewall[11315]: starting server
Dec  9 13:21:31 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:31 VMNode01 dmeventd[11380]: dmeventd ready for processing.
Dec  9 13:21:32 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:33 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:33 VMNode01 lvm[11577]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Dec  9 13:21:34 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:35 VMNode01 pvedaemon[4737]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:35 VMNode01 pve-ha-lrm[10753]: received signal TERM
Dec  9 13:21:35 VMNode01 pve-ha-lrm[10753]: server stopped
Dec  9 13:21:35 VMNode01 pveproxy[28783]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:37 VMNode01 pve-ha-lrm[11675]: starting server
Dec  9 13:21:37 VMNode01 pve-ha-lrm[11675]: status change startup => wait_for_agent_lock
Dec  9 13:21:37 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:38 VMNode01 pve-ha-crm[4742]: received signal TERM
Dec  9 13:21:38 VMNode01 pve-ha-crm[4742]: server received shutdown request
Dec  9 13:21:41 VMNode01 pve-ha-crm[4742]: server stopped
Dec  9 13:21:42 VMNode01 pve-ha-crm[11705]: starting server
Dec  9 13:21:42 VMNode01 pve-ha-crm[11705]: status change startup => wait_for_quorum
Dec  9 13:21:45 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:46 VMNode01 pvedaemon[11764]: send HUP to 4736
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: received signal HUP
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: server closing
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: server shutdown (restart)
Dec  9 13:21:47 VMNode01 pveproxy[11767]: send HUP to 4744
Dec  9 13:21:47 VMNode01 pveproxy[4744]: received signal HUP
Dec  9 13:21:47 VMNode01 pveproxy[4744]: server closing
Dec  9 13:21:47 VMNode01 pveproxy[4744]: server shutdown (restart)
Dec  9 13:21:47 VMNode01 pveproxy[28921]: worker exit
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: restarting server
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4738 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4739 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4737 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: starting 3 worker(s)
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11771 started
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11772 started
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11773 started
Dec  9 13:21:47 VMNode01 pve-ha-crm[11705]: status change wait_for_quorum => slave
Dec  9 13:21:48 VMNode01 spiceproxy[11770]: send HUP to 4758
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: received signal HUP
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: server closing
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: server shutdown (restart)
Dec  9 13:21:48 VMNode01 spiceproxy[4759]: worker exit
Dec  9 13:21:48 VMNode01 pveproxy[30270]: worker exit
Dec  9 13:21:48 VMNode01 pveproxy[4744]: restarting server
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 30270 finished
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 28921 finished
Dec  9 13:21:48 VMNode01 pveproxy[4744]: starting 3 worker(s)
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11777 started
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11778 started
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11779 started
Dec  9 13:21:48 VMNode01 pvestatd[11776]: send HUP to 10540
Dec  9 13:21:48 VMNode01 pvestatd[10540]: received signal HUP
Dec  9 13:21:48 VMNode01 pvestatd[10540]: server shutdown (restart)
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: restarting server
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: worker 4759 finished
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: starting 1 worker(s)
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: worker 11782 started
Dec  9 13:21:49 VMNode01 pvestatd[10540]: restarting server
Dec  9 13:21:58 VMNode01 pveupgrade[30219]: update new package list: /var/lib/pve-manager/pkgupdates
Dec  9 13:22:12 VMNode01 pveproxy[28783]: worker exit
Dec  9 13:22:12 VMNode01 pveproxy[4744]: worker 28783 finished
Dec  9 13:22:19 VMNode01 pvedaemon[11773]: <root@pam> starting task UPID:VMNode01:00003F1F:00087B2F:56688D7B:vncproxy:100:root@pam:
Dec  9 13:22:19 VMNode01 pvedaemon[16159]: starting vnc proxy UPID:VMNode01:00003F1F:00087B2F:56688D7B:vncproxy:100:root@pam:
Dec  9 13:22:40 VMNode01 pvedaemon[11773]: <root@pam> starting task UPID:VMNode01:00003F44:00088370:56688D90:aptupdate::root@pam:
Dec  9 13:22:48 VMNode01 pvedaemon[16196]: update new package list: /var/lib/pve-manager/pkgupdates
Dec  9 13:23:05 VMNode01 pmxcfs[6059]: [main] notice: teardown filesystem
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 rsyslogd-2007: action 'action 17' suspended, next retry is Wed Dec  9 13:23:37 2015 [try http://www.rsyslog.com/e/2007 ]
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: status change slave => wait_for_quorum
Dec  9 13:23:08 VMNode01 pmxcfs[6059]: [main] notice: exit proxmox configuration filesystem (0)
Dec  9 13:23:08 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:18 VMNode01 pve-ha-crm[11705]: status change wait_for_quorum => slave
Dec  9 13:24:06 VMNode01 pveproxy[11778]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:25:01 VMNode01 CRON[16408]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: successfully acquired lock 'ha_manager_lock'
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: watchdog active
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: status change slave => master
 
It seems like a network issue.

How is the physical network set up?

What kind of switch?

is this is a stand alone system or a cluster ?
 
I highly doubt that its a network issue. Everything was working just fine, until one VM acted up and I had to hard STOP it (described it above).
After that, these problems started.

Physical network is basic. Nothing special. Switch is CISCO Catalyst 3750. As I said in my first post, its a standalone system, no cluster.
 
Hi,

I did update earlier, see my earlier post below. The update did not help.

Me again.

I added the pve-no-subscription repo to the list and ran an update. There was a long list and I did a dist-upgrade. Everything was installed fine, however, the problem persist.

Log of the upgrade and I also did a # systemctl restart pve-cluster:
Code:
Dec  9 13:18:08 VMNode01 pvedaemon[30217]: launch command: /usr/bin/vncterm -rfbport 5900 -timeout 10 -authpath /nodes/VMNode01 -perm Sys.Console -notls -listen localhost -c /bin/bash -c pveupgrade --shell
Dec  9 13:18:30 VMNode01 pveproxy[17955]: worker exit
Dec  9 13:18:30 VMNode01 pveproxy[4744]: worker 17955 finished
Dec  9 13:18:30 VMNode01 pveproxy[4744]: starting 1 worker(s)
Dec  9 13:18:30 VMNode01 pveproxy[4744]: worker 30270 started
Dec  9 13:19:41 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:19:41 VMNode01 fusermount[3447]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Dec  9 13:19:42 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:19:50 VMNode01 kernel: [ 5412.473055] vmbr0: port 5(veth201i0) entered disabled state
Dec  9 13:19:50 VMNode01 kernel: [ 5412.474526] device veth201i0 left promiscuous mode
Dec  9 13:19:50 VMNode01 kernel: [ 5412.474530] vmbr0: port 5(veth201i0) entered disabled state
Dec  9 13:21:03 VMNode01 pmxcfs[27001]: [main] notice: teardown filesystem
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:04 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[28921]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pvedaemon[4738]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:05 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:07 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:09 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:10 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:11 VMNode01 pve-ha-crm[4742]: lost lock 'ha_manager_lock - can't create '/etc/pve/priv/lock' (pmxcfs not mounted?)
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:12 VMNode01 pve-ha-lrm[10753]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 systemd[1]: pve-cluster.service stop-sigterm timed out. Killing.
Dec  9 13:21:14 VMNode01 systemd[1]: pve-cluster.service: main process exited, code=killed, status=9/KILL
Dec  9 13:21:14 VMNode01 systemd[1]: Unit pve-cluster.service entered failed state.
Dec  9 13:21:14 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pvecm[6061]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:14 VMNode01 pveproxy[30270]: ipcc_send_rec failed: Connection refused
Dec  9 13:21:15 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:15 VMNode01 lxcfs[6088]: hierarchies: 1: name=systemd
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 2: cpuset
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 3: cpu,cpuacct
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 4: blkio
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 5: memory
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 6: devices
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 7: freezer
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 8: net_cls,net_prio
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 9: perf_event
Dec  9 13:21:15 VMNode01 lxcfs[6088]: 10: hugetlb
Dec  9 13:21:15 VMNode01 kernel: [ 5497.556715] cgroup: new mount options do not match the existing superblock, will be ignored
Dec  9 13:21:16 VMNode01 kernel: [ 5497.827361] audit: type=1400 audit(1449692476.150:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/lxc-start" pid=6096 comm="apparmor_parser"
Dec  9 13:21:16 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:16 VMNode01 lxc-devsetup[6120]: /dev is devtmpfs
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: status change master => lost_manager_lock
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: watchdog closed (disabled)
Dec  9 13:21:17 VMNode01 pve-ha-crm[4742]: status change lost_manager_lock => wait_for_quorum
Dec  9 13:21:22 VMNode01 pve-ha-crm[4742]: status change wait_for_quorum => slave
Dec  9 13:21:28 VMNode01 pvepw-logger[4163]: received terminate request (signal)
Dec  9 13:21:28 VMNode01 pvepw-logger[4163]: stopping pvefw logger
Dec  9 13:21:28 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:28 VMNode01 pvepw-logger[11250]: starting pvefw logger
Dec  9 13:21:28 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: received signal TERM
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: server closing
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: clear firewall rules
Dec  9 13:21:29 VMNode01 pve-firewall[10904]: server stopped
Dec  9 13:21:30 VMNode01 pve-firewall[11315]: starting server
Dec  9 13:21:31 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:31 VMNode01 dmeventd[11380]: dmeventd ready for processing.
Dec  9 13:21:32 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:33 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:33 VMNode01 lvm[11577]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Dec  9 13:21:34 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:35 VMNode01 pvedaemon[4737]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:35 VMNode01 pve-ha-lrm[10753]: received signal TERM
Dec  9 13:21:35 VMNode01 pve-ha-lrm[10753]: server stopped
Dec  9 13:21:35 VMNode01 pveproxy[28783]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:21:37 VMNode01 pve-ha-lrm[11675]: starting server
Dec  9 13:21:37 VMNode01 pve-ha-lrm[11675]: status change startup => wait_for_agent_lock
Dec  9 13:21:37 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:38 VMNode01 pve-ha-crm[4742]: received signal TERM
Dec  9 13:21:38 VMNode01 pve-ha-crm[4742]: server received shutdown request
Dec  9 13:21:41 VMNode01 pve-ha-crm[4742]: server stopped
Dec  9 13:21:42 VMNode01 pve-ha-crm[11705]: starting server
Dec  9 13:21:42 VMNode01 pve-ha-crm[11705]: status change startup => wait_for_quorum
Dec  9 13:21:45 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:21:46 VMNode01 pvedaemon[11764]: send HUP to 4736
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: received signal HUP
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: server closing
Dec  9 13:21:46 VMNode01 pvedaemon[4736]: server shutdown (restart)
Dec  9 13:21:47 VMNode01 pveproxy[11767]: send HUP to 4744
Dec  9 13:21:47 VMNode01 pveproxy[4744]: received signal HUP
Dec  9 13:21:47 VMNode01 pveproxy[4744]: server closing
Dec  9 13:21:47 VMNode01 pveproxy[4744]: server shutdown (restart)
Dec  9 13:21:47 VMNode01 pveproxy[28921]: worker exit
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: restarting server
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4738 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4739 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 4737 finished
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: starting 3 worker(s)
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11771 started
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11772 started
Dec  9 13:21:47 VMNode01 pvedaemon[4736]: worker 11773 started
Dec  9 13:21:47 VMNode01 pve-ha-crm[11705]: status change wait_for_quorum => slave
Dec  9 13:21:48 VMNode01 spiceproxy[11770]: send HUP to 4758
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: received signal HUP
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: server closing
Dec  9 13:21:48 VMNode01 spiceproxy[4758]: server shutdown (restart)
Dec  9 13:21:48 VMNode01 spiceproxy[4759]: worker exit
Dec  9 13:21:48 VMNode01 pveproxy[30270]: worker exit
Dec  9 13:21:48 VMNode01 pveproxy[4744]: restarting server
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 30270 finished
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 28921 finished
Dec  9 13:21:48 VMNode01 pveproxy[4744]: starting 3 worker(s)
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11777 started
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11778 started
Dec  9 13:21:48 VMNode01 pveproxy[4744]: worker 11779 started
Dec  9 13:21:48 VMNode01 pvestatd[11776]: send HUP to 10540
Dec  9 13:21:48 VMNode01 pvestatd[10540]: received signal HUP
Dec  9 13:21:48 VMNode01 pvestatd[10540]: server shutdown (restart)
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: restarting server
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: worker 4759 finished
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: starting 1 worker(s)
Dec  9 13:21:49 VMNode01 spiceproxy[4758]: worker 11782 started
Dec  9 13:21:49 VMNode01 pvestatd[10540]: restarting server
Dec  9 13:21:58 VMNode01 pveupgrade[30219]: update new package list: /var/lib/pve-manager/pkgupdates
Dec  9 13:22:12 VMNode01 pveproxy[28783]: worker exit
Dec  9 13:22:12 VMNode01 pveproxy[4744]: worker 28783 finished
Dec  9 13:22:19 VMNode01 pvedaemon[11773]: <root@pam> starting task UPID:VMNode01:00003F1F:00087B2F:56688D7B:vncproxy:100:root@pam:
Dec  9 13:22:19 VMNode01 pvedaemon[16159]: starting vnc proxy UPID:VMNode01:00003F1F:00087B2F:56688D7B:vncproxy:100:root@pam:
Dec  9 13:22:40 VMNode01 pvedaemon[11773]: <root@pam> starting task UPID:VMNode01:00003F44:00088370:56688D90:aptupdate::root@pam:
Dec  9 13:22:48 VMNode01 pvedaemon[16196]: update new package list: /var/lib/pve-manager/pkgupdates
Dec  9 13:23:05 VMNode01 pmxcfs[6059]: [main] notice: teardown filesystem
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11777]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11772]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11773]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pvedaemon[11771]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 rsyslogd-2007: action 'action 17' suspended, next retry is Wed Dec  9 13:23:37 2015 [try http://www.rsyslog.com/e/2007 ]
Dec  9 13:23:07 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:07 VMNode01 pve-ha-lrm[11675]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pve-ha-crm[11705]: status change slave => wait_for_quorum
Dec  9 13:23:08 VMNode01 pmxcfs[6059]: [main] notice: exit proxmox configuration filesystem (0)
Dec  9 13:23:08 VMNode01 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pveproxy[11779]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:08 VMNode01 pvecm[16257]: ipcc_send_rec failed: Connection refused
Dec  9 13:23:18 VMNode01 pve-ha-crm[11705]: status change wait_for_quorum => slave
Dec  9 13:24:06 VMNode01 pveproxy[11778]: ipcc_send_rec failed: Transport endpoint is not connected
Dec  9 13:25:01 VMNode01 CRON[16408]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: successfully acquired lock 'ha_manager_lock'
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: watchdog active
Dec  9 13:25:10 VMNode01 pve-ha-crm[11705]: status change slave => master
 
Last edited:
No, it shows as "dead".
I tried starting it, but it does not change. It will show as started on command line, but marks it as "inactive"
 
Hi,

I still have the same error as before. Any ideas?

quorum.PNGProxServices.PNG

The VM conf:
Code:
root@VMNode01:/etc/pve/nodes/VMNode01/qemu-server# cat 100.conf
#Main Server running Windows Small Business 2011 Std.
#
#Server has AD, DHCP, DNS roles.
boot: cdn
bootdisk: scsi0
cores: 2
hotplug: network,usb,disk
ide2: none,media=cdrom
memory: 32768
name: WinSBS2011Std
net0: virtio=2E:6E:6E:05:F8:20,bridge=vmbr0
onboot: 1
ostype: win7
parent: BackupFail
scsi0: VM-Storage:vm-100-disk-1,size=84G
scsi1: VM-Storage:vm-100-disk-2,size=32G
scsihw: virtio-scsi-pci
sockets: 2
startup: order=1,up=240,down=180
vga: qxl

And the error again:
Code:
Dec 10 09:18:16 VMNode01 pvestatd[21515]: starting server
Dec 10 09:18:20 VMNode01 pmxcfs[21236]: [main] notice: teardown filesystem
Dec 10 09:18:21 VMNode01 pve-ha-lrm[4741]: ipcc_send_rec failed: Transport endpoint is not connected
Dec 10 09:18:21 VMNode01 pve-ha-lrm[4741]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:21 VMNode01 pve-ha-lrm[4741]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:21 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Transport endpoint is not connected
Dec 10 09:18:21 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:21 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:21 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pveproxy[21487]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pvedaemon[21466]: ipcc_send_rec failed: Transport endpoint is not connected
Dec 10 09:18:22 VMNode01 pvedaemon[21466]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pvedaemon[21466]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:22 VMNode01 pmxcfs[21236]: [main] notice: exit proxmox configuration filesystem (0)
Dec 10 09:18:23 VMNode01 pvecm[21531]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:23 VMNode01 pvecm[21531]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:23 VMNode01 pvecm[21531]: ipcc_send_rec failed: Connection refused
Dec 10 09:18:28 VMNode01 pve-ha-crm[4732]: ipcc_send_rec failed: Transport endpoint is not connected

Thanks.
 
After rebooting the node, the VM did start, however, the ipcc_send error will still come up.
 
That is correct, just one physical server and one pro mix install. No cluster at all.


Sent from my iPhone using Tapatalk
 
And the error again:
Code:
Dec 10 09:18:20 VMNode01 pmxcfs[21236]: [main] notice: teardown filesystem
Dec 10 09:18:22 VMNode01 pmxcfs[21236]: [main] notice: exit proxmox configuration filesystem (0)
Dec 10 09:18:23 VMNode01 pvecm[21531]: ipcc_send_rec failed: Connection refused

The question is why pmxcfs gets stopped?
 
The syslog does not show. Is there a way for me to check, or to change the level of logging?

I also have the issue, that the server reboots, without any notice or warning (maybe for another thread), but changing the level of logging to log more details, might help to find the ipcc problem and why the server restarts (looks like kernel panic, but not sure).

Thanks.
 
Same trouble. Sometimes on vm with Windows 7 do rollback from snapshot and got:

Failed to create message: Input/output error
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit ...

At second attempt - vm rollback fine. May be any timeouts or queues?

pve-manager/4.1-1/2f9650d4 (running kernel: 4.2.6-1-pve)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!