iowait when pve services down

akempiak

New Member
Sep 14, 2016
3
0
1
40
Hi,
On my proxmox cluster, every nodes were red in the webUI, like if something went wrong with multicast, anyway every VMs were OK.
I recovered the cluster like this on each server :
killall -9 corosync
systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pvestatsd

But here is the REALLY strange thing. The storage is a NFS Netapp. For several virtuals machines on a proxmox server, there were huges iowait EXACTLY during this cluster services outage.
Can you help please ?
Regards.
 
Last edited:
root@kvm14:~# pveversion -v
proxmox-ve: 4.2-58 (running kernel: 4.4.8-1-pve)
pve-manager: 4.2-17 (running version: 4.2-17/e1400248)
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve2
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-43
qemu-server: 4.0-85
pve-firmware: 1.1-8
libpve-common-perl: 4.0-71
libpve-access-control: 4.0-18
libpve-storage-perl: 4.0-56
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6-1
pve-container: 1.0-71
pve-firewall: 2.0-29
pve-ha-manager: 1.0-33
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80
 
For several virtuals machines on a proxmox server, there were huges iowait EXACTLY during this cluster services outage.
iowait is a tricky metric, because it means "% of cpu time where the cpu is idle AND waits for io" meaning high iowait with low cpu usage can be no problem at all and low iowait with full cpu usage can mean a very slow machine...
 
Physical server has 2 CPU Intel Xeon E5-2687W v3, one of the most powerful CPU on the market.
The VM has 8 vCPU. The storage behind (Netapp) was not over-used. Here is how iowaits looked like, appeared as soon as pve cluster services failed, and then disapperead when services restored :

upload_2016-9-16_10-30-19.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!