ZFS poor performance when high IO load on one pool

[mauriceklein]

Hi there,

i have an issue with zfs on Proxmox VE 8.1.3

I have 3 zfs pools, one mirror consisting of 2 nvme ssd, one raidz1 consisting of 4 sata hdd and one raidz1 consisting of 4 sata hdd.
Whenever there's high io load on of my hdd pools from one vm i get an io delay of about 60% in the overview and a cpu usage of around 5%.

Now the wired thing is that other vm's that only run on the nvme mirror get completly unusable even though there is no load on that pool and the same goes for the other hdd pool.

How can the IO on one pool slow down all other pools?
Also can't be the sata controller since the nvme drives use different pcie lanes directly to the CPU.
The server has an EPYC 24 core cpu and 512GB memory.
Arc cache is using half of that and total usage is 360.

 

[jsterr]

What devices are you using for the pools? Please provide vendor and model.

 

[mauriceklein]

2 Curical P5+ 2TB as ZFS Mirror
4 Toshiba MG09 18TB Enterprise capacity as ZFS Raidz
4 WD Red 3TB as ZFS Raidz

The high IO Load is on the 4 18TB pool and that slows even the nvme ssd pool down to the point that it's unusable.

 

[mauriceklein]

Hi there,

any news on this?

 

[NiceRath]

Maybe related to: https://forum.proxmox.com/threads/upgrade-to-8-1-3-increased-io-delay-by-far.138644/

 

[Nemesiz]

All ZFS pools in the same host share the same ZFS memory. I don't know are you effected read (ARC) or write (dirty cache) cache. I can suggest to lower dirty cache or change zfs_txg_timeout. How will it help for you or will it help at all I don't know.