Proxmox High IO Delay

[JWilliamson]

I've been using proxmox for a few weeks and have a total of 13 active CTs/VMs, all with various things running, nothing high load (peak cpu usage is 8%)

I noticed it was taking a while for everything to load, and saw my IO delay has been hovering around 20-30, sometimes spiking above 50.

Proxmox is installed on a dell r720 across 2x 500gb 5400 RPM hard drives. With 4x 1TB 5400RPM hard drives, set up in raid-z1 for vms/cts

All drives are connected to a hba card in IT mode and passed straight through to proxmox.

Rest of my specs are:
Proxmox v7.2-11
64gb ecc ram
2x Intel Xeon E5-2690 @ 2.90Ghz
pve no-subscription-updates enabled.

Any ideas as to why my io delay is this high? Maybe something is configured wrong?

Any help is appreciated,
TIA.

 

[leesteken]

You have slow drives and they are not in the most performant configuration, but your data is more safe this way.
Note that this is a community forum with volunteers, if you do want urgent and specialized support, I suggest you buy a subscription for guaranteed response times.

 

[JWilliamson]

You have slow drives and they are not in the most performant configuration, but your data is more safe this way.
Note that this is a community forum with volunteers, if you do want urgent and specialized support, I suggest you buy a subscription for guaranteed response times.

Would upgrading to 7400 rpm drives or maybe faster sas drives in the future reduce it?

 

[Dunuin]

Running multiple guests creates alot of IO and HDDs in general are terrible at IOPS performance. Thats why you don't install your single bare metal OSs on HDDs for over a decade. Running 13 OSs of 2 HDDs doesn't make it better...I would really recommend to get some SSDs...at least for the VM/LXC storage. A good SSD is easily 100x faster.
And IOPS performance will only scale with the number of vdevs, not the number of disks. 4 disks in a raidz1 got the same IOPS performance as just a single disk. Better would be a striped mirror (raid10) where you got 2 vdevs, so double the IOPS performance.

And don't cheap out on SSDs...QLC consumer SSDs can be as slow as HDDs...especially when used with ZFS.

 

[JWilliamson]

Running multiple guests creates alot of IO and HDDs in general are terrible at IOPS performance. Thats why you don't install your bare metal OSs on HDDs for over a decade. Running 13 OSs of 2 HDDs doesn't make it better...I would really recommend to get some SSDs...at least for the VM/LXC storage. A good SSD is easily 100x faster.
And IOPS performance will only scale with the number of vdevs, not the number of disks. 4 disks in a raidz1 got the same IOPS performance as just a single disk. Better would be a striped mirror (raid10) where you got 2 vdevs, so double the IOPS performance.

And don't cheap out on SSDs...QLC consumer SSDs can be as slow as HDDs...especially when used with ZFS.

Thanks for the reply.

Probably should of clarified, all vm's/ct's are on the raid z1 array. The 2x 500gb drives are just for proxmox boot.

So if i replace the 4x hdd's to say 6x 1tb ssd's in raid 10 (match my current capacity), that would fix it?
If i kept the 2x 500gb hdds for boot, would this still cause problems? Or would it be best to replace these aswell?

 

[Dunuin]

HDDs are fine for PVE. It doesn't need much performance. Problem is usually just the VM/LXC storage.
And yes, SSDs should solve that problem. But keep in mind that it is highly recommended to use enterprise grade SSDs with powerloss protection when using ZFS. And I would guess you could even use 4 SSDs in a raidz1 and it might still be fast enough. 6 disk in a striped mirror of cause would still tripple the IOPS performance, but maybe you don't need that much performance, as the IOPS performance should be magnitues better even with a raidz1.
Also keep in mind that raidz needs a big volblocksize or you are wasting alot of capacity. 4 disks (with 4k sectors) in a raidz1 would need a volblocksize of at least 16K and 6 disk in a raidz1 at least 32K. And performance of IO that is smaller than your volblocksize will be terrible. So really bad idea to use a raidz1 of 3 or more disks with PosgrSQL (doing 8K writes) and raidz1 of 5 or more disks with with MySQL (doing 16K writes).