[SOLVED] High IO Delay

W

Whattteva

Member
Feb 16, 2023
29
9
8
Hello, I have a high IO delay issue reaching a peak of 25% that has been bothering me for quite some time. I've seen a few threads about it but none of them really solved my issue.

I've noticed that I experience it under these conditions:
  • Copying/moving files within a Windows 11 VM.
  • Installing MacOS in a VM.
  • Cloning a VM through Proxmox web admin.
In all of these cases, IO delay coincides with high Disk IO and CPU usage does not seem to be of relevance. Any help is appreciated to at least point me in the right direction to a resolution.

My specs are as follows:
  • Supermicro X11SPI-TF
  • Intel Xeon Silver 4210T (10c/20t) Cascade Lake 2.3/3.2 GHz 95 W
  • 224 GB DDR4 2400 ECC LRDIMM
  • 2x Inland Professional 512 GB SSD - Mirrored
  • Proxmox VE 7.3-3
If more details is needed, let me know.
 
Can you provide more information about your disk setup? Are you using the SSD mirror both for PVE and as storage for your VMs? A simple mirror of SSDs does not provide much performance when multiple VMs are running on the same storage and heavy I/O operations are running.
 
They're setup just using whatever the default Proxmox installation does, nothing custom. Also yes, I am using them for both Proxmox and the VM storage.

I do run about 6 VM's, but all of them are basically idle. Whenever I get these IO delay events, I can trace it to exactly either something disk heavy I'm doing in 1 VM (ie. file copy) or I'm cloning a VM on the Proxmox UI.

I understand that 1 mirror isn't all that fast, but I find it odd that just a simple file copy in 1 VM could slow down everything to a crawl (including other VM's).
 
Last edited:
Just a first guess:
Is any component overheating and throttling on higher temps? 'Action' like copying pushes just over the limit?
Is in the BIOS anything to ASPM related active? If yes, try disabling it.
Are your vms configured with virtio-disk-controller or scsi-single? (if sata that's not really fast)
Try writeback-cache in your vms, instead of off.
 
mr44er said:
Just a first guess:
Is any component overheating and throttling on higher temps? 'Action' like copying pushes just over the limit?
Click to expand...
Unlikely, IPMI isn't giving me any alerts and CPU usage is low besides disk activity.

mr44er said:
Is in the BIOS anything to ASPM related active? If yes, try disabling it.
Click to expand...
Not sure about this one. I left most things on default. I'd like to check it but it would require me to take the server down, which isn't really feasible right now, but I will look into this.

mr44er said:
Are your vms configured with virtio-disk-controller or scsi-single? (if sata that's not really fast)
Click to expand...
They're all set as VirtIO SCSI Single and they have the VirtIO guest drivers installed.

mr44er said:
Try writeback-cache in your vms, instead of off.
Click to expand...
No writeback cache. Could this be the issue? Is it safe to use the cache?
 
How mirrored drive ?
zfs ? hw raid ?

your ssd are cheap so you can't really avoid high io delay.
imo, best perf will be with one ssd for ext4 ( pve will use lvmthin to store vDisks) , then the other ssd as an ext4 disk to provide datastore for an install of pbs alongside pve, to daily backup.
write cache option in each vm need to be enabled to get best perf but lost data is possible if power failure.

... sorry for my english...
 
Whattteva said:
Unlikely, IPMI isn't giving me any alerts and CPU usage is low besides disk activity.
Click to expand...
It is not only the cpu that can overheat, also the NIC, disk-controller, anything with a heat sink and not everything has a sensor to view remotely or detect at all.


Whattteva said:
No writeback cache. Could this be the issue? Is it safe to use the cache?
Click to expand...
Could be or anything with these SSDs. I don't know them and can't compare how they behave/perform under best circumstances.
It should be safe for a test to try writeback to rule that out, but not for longtime/production in this constellation.
Maybe try with the win11-vm only, if it performs better etc.

_gabriel said:
your ssd are cheap so you can't really avoid high io delay.
Click to expand...
Yup, just found them on amazon.
They don't perform good enough and the IO-delay starts exactly when the internal cache of these SSDs is reached (has nothing to do with the setting of the VM, that's another cache and as gabriel said it is unsafe and you will loose data on power failure if not corrupt the whole disks in the end)
 
Last edited:
_gabriel said:
How mirrored drive ?
zfs ? hw raid ?
Click to expand...
ZFS.

_gabriel said:
your ssd are cheap so you can't really avoid high io delay.
Click to expand...
Yeah, I'm aware that they're cheap SSD's. But, I just didn't think it would make such a significant IO delay just from a simple file copy within one VM.
I'm definitely open to replace them, but I just want to make sure that is indeed the root issue before going out and buying new SSD's just to discover that it doesn't solve the issue.

_gabriel said:
write cache option in each vm need to be enabled to get best perf but lost data is possible if power failure.
Click to expand...
Yeah, this is why I'm not too keen on enabling it.
 
mr44er said:
It is not only the cpu that can overheat, also the NIC, disk-controller, anything with a heat sink and not everything has a sensor to view remotely or detect at all.
Click to expand...
I understand that, but the chassis is massive (Fractal Design Define 7 XL) with plenty of space and no expansion cards other than a SAS2 HBA. It also has 6x 140mm fans so it's well-ventilated.

mr44er said:
Could be or anything with these SSDs. I don't know them and can't compare how they behave/perform under best circumstances.
It should be safe for a test to try writeback to rule that out, but not for longtime/production in this constellation.
Maybe try with the win11-vm only, if it performs better etc.
Click to expand...
Thanks, I will be trying this out.

mr44er said:
Yup, just found them on amazon.
They don't perform good enough and the IO-delay starts exactly when the internal cache of these SSDs is reached (has nothing to do with the setting of the VM, that's another cache and as gabriel said it is unsafe and you will loose data on power failure if not corrupt the whole disks in the end)
Click to expand...
Hmm, understood. If the writeback cache experiment proves unfruitful, I will be looking into getting another pair of SSD's.
 
  • Like
Reactions: BAGUS
Whattteva said:
but I just want to make sure that is indeed the root issue before going out and buying new SSD's just to discover that it doesn't solve the issue.
Click to expand...
So for only this short test it is ok to try writeback with one vm+reboot. If it's slightly better, then you know for sure.

Anyway I think the device has it enabled itself hardcoded (maybe changeable, but reverts on reboot). Try on the host: smartctl -x /dev/sdX | grep Writeback

Disabled = on the safe side
Enabled = data loss/corruption for sure on power loss
 
Last edited:
zfs is the worst case, please read forums topic, you'll get fired because use zfs on consumer ssd...
zfs is for entreprise storage disks + without a hw raid controller.
 
Last edited:
_gabriel said:
zfs is the worst case, please read forums topic, you'll get fired because use zfs on consumer ssd...
zfs is for entreprise storage disks + without a hw raid controller.
Click to expand...
Jup. And that mainboard+CPU+RAM costs over 2000€ but you were only willing to spend 74€ on those SSD. Thats saving bucks at the wrong end.

Whattteva said:
Care to link me the thread for reference?
Click to expand...
Also see the ZFS Benchmark Papers FAQ:
https://www.proxmox.com/de/downloads/item/proxmox-ve-zfs-benchmark-2020
Can I use consumer or pro-sumer SSDs, as these are much cheaper than enterprise-class SSDs?
No. Never. These SSDs wont provide the required performance, reliability or endurance. See the fio results from before and/or run your own fio tests.
Click to expand...
1676577731691.png
So your SSDs are probably somewhere in the range of that Crucial MX100.



And for ZFS on top of HW raid controllers see the OpenZFS documentation:
https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hardware-raid-controllers
 
Last edited:
  • Like
Reactions: mow
Dunuin said:
Jup. And that mainboard+CPU+RAM costs over 2000€ but you were only willing to spend 74€ on those SSD. Thats saving bucks at the wrong end.
Click to expand...
Haha, I wasn't trying to save money or anything. I'm totally fine with buying new enterprise SSD's if need be. Those SSD's just happened to be something I already had on hand and didn't have to wait for shipping etc. to use. And really, for the most part, they've been totally fine other than the occasional spikes of IO delay (I've been using them for about 3 months).

Dunuin said:
Also see the ZFS Benchmark Papers FAQ:
https://www.proxmox.com/de/downloads/item/proxmox-ve-zfs-benchmark-2020

View attachment 46917
So your SSDs are probably somewhere in the range of that Crucial MX100.
Click to expand...
Ah thank you for this. I will be ordering another pair of SSD's. Any recommendations for decently-priced ones?
 
Whattteva said:
Any recommendations for decently-priced ones?
Click to expand...
https://yourcmc.ru/wiki/Ceph_performance recommends:
Micron 5100/5200, 9300. Maybe 5300, 7300 too
Seagate Nytro 1351/1551
HGST SN260
Intel P4500

There is also an excel list with benchmarks on more disks.
Ceph is more demanding on SSDs and what's good for ceph is also good for ZFS.
Besides that the link is a good lecture with explaining the different quality of SSDs (just ignore the ceph-theming :) )
 
  • Like
Reactions: herzkerl, baudneo, mow and 1 other person
Whattteva said:
I've ordered a pair of Intel DC S3500 480 GB. Those should be OK, I presume?
Click to expand...
Not the greatest write performance and durability as they are designed for read-intense workloads but yes, should perform way better than the consumer disks you got right now.
 
You must log in or register to reply here.
Top Bottom