Question mark and "proxy detected vanished client connection" error

[michael_hess]

I have four nodes, three are old HP server, the 4th is a new Dell server.

I just added the Dell to the cluster, it seemed fine for a few days, but now I'm getting storage disconnects and the proxy detected vanished client connection errors. The GUI is slow/buggy on that server as well.

I can't migrate off the vm's to reboot the server, and the VM's are acting strangely, probably due to the storage going offline randomly. Here's some recent logs:

Feb 26 10:47:52 proxmox-srv3-n0 pvestatd[1360217]: storage 'DPS500T-a' is not online
Feb 26 10:48:02 proxmox-srv3-n0 pvestatd[1360217]: storage 'IF-InfoTech' is not online
Feb 26 10:48:04 proxmox-srv3-n0 pvestatd[1360217]: storage 'Rackstation' is not online
Feb 26 10:48:04 proxmox-srv3-n0 nfsidmap[1360325]: nss_getpwnam: name 'nobody' does not map into domain 'nols.edu'
Feb 26 10:48:04 proxmox-srv3-n0 nfsidmap[1360326]: nss_name_to_gid: name 'nobody' does not map into domain 'nols.edu'
Feb 26 10:48:19 proxmox-srv3-n0 pveproxy[1357538]: proxy detected vanished client connection
Feb 26 10:48:44 proxmox-srv3-n0 pveproxy[1358461]: proxy detected vanished client connection
Feb 26 10:50:16 proxmox-srv3-n0 pmxcfs[169297]: [status] notice: received log
Feb 26 10:51:28 proxmox-srv3-n0 pveproxy[1038073]: proxy detected vanished client connection
Feb 26 10:51:28 proxmox-srv3-n0 pveproxy[1357538]: proxy detected vanished client connection
Feb 26 10:51:31 proxmox-srv3-n0 pveproxy[1357538]: proxy detected vanished client connection
Feb 26 10:51:33 proxmox-srv3-n0 pveproxy[1358461]: proxy detected vanished client connection
root@proxmox-srv3-n0:~# systemctl restart pvestatd
Feb 26 11:01:18 proxmox-srv3-n0 pveproxy[1358461]: proxy detected vanished client connection
Feb 26 11:01:22 proxmox-srv3-n0 pve-ha-crm[1508]: cluster had no service configured for 90 rounds, going idle.
Feb 26 11:01:22 proxmox-srv3-n0 pve-ha-crm[1508]: watchdog closed (disabled)
Feb 26 11:01:22 proxmox-srv3-n0 pve-ha-crm[1508]: status change master => wait_for_quorum
Feb 26 11:01:22 proxmox-srv3-n0 pveproxy[1357538]: proxy detected vanished client connection
Feb 26 11:01:25 proxmox-srv3-n0 pveproxy[1357538]: proxy detected vanished client connection
Feb 26 11:01:48 proxmox-srv3-n0 pveproxy[1358461]: proxy detected vanished client connection
Feb 26 11:01:51 proxmox-srv3-n0 pveproxy[1358461]: proxy detected vanished client connection

As you can see, I've tried restarting pvestatd, I was able to migrate one VM before the issue came back, with that trick. But now, 2 vm's remain and I can't get anything but connection error - timeout messages in the GUI.

Any thoughts? My storage is all on a dedicated vlan, iscsi and nfs to multiple arrays. The 3 HP nodes are all working fine.