iowait when pve services down

[akempiak]

Hi,
On my proxmox cluster, every nodes were red in the webUI, like if something went wrong with multicast, anyway every VMs were OK.
I recovered the cluster like this on each server :
killall -9 corosync
systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pvestatsd

But here is the REALLY strange thing. The storage is a NFS Netapp. For several virtuals machines on a proxmox server, there were huges iowait EXACTLY during this cluster services outage.
Can you help please ?
Regards.

 

[fireon]

Please write here the version.

pveversion -v

 

[akempiak]

root@kvm14:~# pveversion -v
proxmox-ve: 4.2-58 (running kernel: 4.4.8-1-pve)
pve-manager: 4.2-17 (running version: 4.2-17/e1400248)
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve2
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-43
qemu-server: 4.0-85
pve-firmware: 1.1-8
libpve-common-perl: 4.0-71
libpve-access-control: 4.0-18
libpve-storage-perl: 4.0-56
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6-1
pve-container: 1.0-71
pve-firewall: 2.0-29
pve-ha-manager: 1.0-33
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80

 

[dcsapak]

For several virtuals machines on a proxmox server, there were huges iowait EXACTLY during this cluster services outage.

iowait is a tricky metric, because it means "% of cpu time where the cpu is idle AND waits for io" meaning high iowait with low cpu usage can be no problem at all and low iowait with full cpu usage can mean a very slow machine...

 

[akempiak]

Physical server has 2 CPU Intel Xeon E5-2687W v3, one of the most powerful CPU on the market.
The VM has 8 vCPU. The storage behind (Netapp) was not over-used. Here is how iowaits looked like, appeared as soon as pve cluster services failed, and then disapperead when services restored :