High IO Delay

[vmbuff]

I am getting High IO delay. What I have done so far is

- increased ram from 12GB to 24GB
- reduced SWAP to 10 since before it kept writing on where my proxmox was installed which is a slow HDD
- changed my Windows VM to not use ram ballooning by setting max and min memory to be the same in Proxmox

I am still seeing spikes in my IO Delay. Screenshot of the system and see the spike in IO delay. And screenshot of the iotop. I noticed that when IO delay spikes, there's a bunch of these zvol that pops up. I don't know what that is.

I am using SSD for my drives.

Thoughts on what I can do? Thanks

 

[Dunuin]

What SSD are you using? Consumer SSDs can't handle that much IO, especially if there are sync writes because the SSD can't cache them without build in powerloss protection. And if I look at your disk writes I would bet you got alot of write amplification. So looks like your SSD is struggeling with ZFS doing alot of IOPS.

You could run fio to benchmark the performance of your disks and compare it to the results of the proxmox zfs benchmark paper: https://proxmox.com/en/downloads/item/proxmox-ve-zfs-benchmark-2020

iostat (sysstat package) is also useful to monitor the IOPS and data written/read.

 

[vmbuff]

Ah yes I am using a consumer SSD and especially for this SSD, I am using a really cheap SSD (Inland Professional) which is slower than those "gaming" or faster consumer SSD.

Thank you! I'll dig into this.

 

[vmbuff]

I didn't get a chance to run the tests but I migrated the VM off the Inland Professional SSD over to a Samsung 870 EVO and I don't have any more IO Delay issues. Thanks again for the help.

 

[nvasilev]

I also had high IO delay on a Thinkpad T430 laptop with a SILICON POWER Ace A55 256gb SSD. After seeing that @vmbuff had success with the Samsung SSD I tried with a Samsung 850 Pro 256GB SSD that I had and so far the results are great!

Before I had 25-35% IO delay and now it is under 1%, rarely goes to 4-5%.