Increased IO Delay radically, how to reduce?

[fintnovo]

Good day.
3 month ago average IO Delay increased dramatically, about 10 or 15 times. Nothing radically changed in VMs. Hypervisor hardware also the same. I have 17 VM running: (10 with Windows 7,10, Server 2012) and (7 with Linux Ubuntu 18.04).
There are 3 disks also:

all VMs on nvme0n1 (m.2 ssd)
sda - only backups (sata hdd)
sdb - PVE + swap (sata ssd)
Here is screenshot with typical server load (avarage IO Delay about 10%)
Free memory about 40%. Average CPU usage about 20% ....

Any idea?

 

[PeterH896]

Hi,

you could use "iotop" to see which process is causing this IO delay.

 

[Whatever]

If you are making backups with build-in function have you tuned "vm.dirty_background_bytes" and "vm.dirty_bytes"?

More info here: https://forum.proxmox.com/threads/io-performance-tuning.15893/

 

[lhorace]

You also want to look at the total IOPS when the storage is NOT busy

 

[fintnovo]

Hi,

you could use "iotop" to see which process is causing this IO delay.

Good day, Peter/
Here is iotop output:

 

[fintnovo]

If you are making backups with build-in function have you tuned "vm.dirty_background_bytes" and "vm.dirty_bytes"?

More info here: https://forum.proxmox.com/threads/io-performance-tuning.15893/

Thanks, I saw this post. I do some modifications in sysctl.conf and reboot, but nothing changed:

 

[fintnovo]

You also want to look at the total IOPS when the storage is NOT busy

I shutdown all VMs, then run "iotops", here is output:

also here is average hourly output with disabled VMs (IO Delay reduced..):

 

[Whatever]

Thanks, I saw this post. I do some modifications in sysctl.conf and reboot, but nothing changed:
View attachment 10051

_ratio parameters depend on total RAM. Thus it's recommended to use _bytes instead
a

one more thing to be clarified. Could you print the output of "zpool status" command (there is l2arc_feed process on one of your screenshots)

 

[fintnovo]

 

[AxelTwin]

If you are making backups with build-in function have you tuned "vm.dirty_background_bytes" and "vm.dirty_bytes"?

More info here: https://forum.proxmox.com/threads/io-performance-tuning.15893/

Is this still valid since the postis from 2013 ?