High IO.Delay (15-30%) doing any disk related work on node

[Tera-sat]

Hi, I'm new to ProxMox VE, worked professionally with VmWare for many years.
I have a small homelab running on a Dell T320 with a raid 5 array on 7 disks (+ one spare)
ProxMox itself is installed on a samsung SSD, 250gb
Every time I make a snapshot or run a backup I get IO delays in the region of 15-30-40% and everything slows down to a crawl

My old VmWare ESXi 7 ran on the exact same setup and had no problem with IO delay what so ever.....

If ProxMox wan't to be a serious alternative to VmWare, HyperV, and Citrix then you really have to run on same hardware without any problem.....
I really like ProxMox, don't get me wrong but this has to be fixed.....

 

[leesteken]

I have a small homelab running on a Dell T320 with a raid 5 array on 7 disks (+ one spare)

Just to be clear: do you use the same hardware RAID5 controller with Proxmox as ESXi or are you using ZFS RAIDz1 (which tends to surprise people by behaving differently)?

Every time I make a snapshot or run a backup I get IO delays in the region of 15-30-40% and everything slows down to a crawl

Proxmox tends to use a lot of threads for backups, which might cause many of them to wait due to slow write performance and therefore show a high IO delay. This was a problem in the past and I'm not sure how change this setting. I'll might do a search later.

 

[Tera-sat]

I use the excact same hardware and setup as I did with VmWare.
Perc h310 controller with 8, 3Tb Seagate Constellation disks setup in a raid 5 with one hotspare. ProxMox is installed on a 250Gb Samsung SSD
The raid is configured in PM as a storage pool formated in Ext4 to store disk files

 

[leesteken]

Perc h310 controller with 8, 3Tb Seagate Constellation disks setup in a raid 5 with one hotspare.

Thank you for explaining this and definitely ruling out slowness due to ZFS overhead and write amplification.

The raid is configured in PM as a storage pool formated in Ext4 to store disk files

Using LVM thin to store virtual disks will most likely be faster and give less IO delay than using files for virtual disks.

 

[Tera-sat]

Thank you for explaining this and definitely ruling out slowness due to ZFS overhead and write amplification.

Using LVM thin to store virtual disks will most likely be faster and give less IO delay than using files for virtual disks.

The data store IS LVM thin

 

[_gabriel]

Proxmox VE running Linux Kernel has all cpu mitigations enabled where ESXi not.
Try with mitigations=off as kernel options.

edit : https://forum.proxmox.com/threads/ubuntu-20-vm-much-slower-than-in-esxi.149444/post-677546

 

[Tera-sat]

Proxmox VE running Linux Kernel has all cpu mitigations enabled where ESXi not.
Try with mitigations=off as kernel options.

Hmmmm. And how do I do that ?

 

[_gabriel]

https://forum.proxmox.com/threads/ubuntu-20-vm-much-slower-than-in-esxi.149444/post-677546

 

[Tera-sat]

https://forum.proxmox.com/threads/ubuntu-20-vm-much-slower-than-in-esxi.149444/post-677546

Tried that, had no effect what so ever ........

 

[spirit]

for backup, try to enable the need fleecing backup option.

without it, if your backup storage is slow, it can slowdown your production vm when backup is running. (because if your vm os try to write on a block not yet backuped, it need to be send first to the backup storage).

The fleecing backup option is like a write cache in front of your backup storage.

 

[leesteken]

Proxmox tends to use a lot of threads for backups, which might cause many of them to wait due to slow write performance and therefore show a high IO delay.

I found the option for reducing backup workers: https://forum.proxmox.com/threads/high-i-o-wait-on-nodes-during-backups.150548/post-680987