IO Delay, configured wrong?

[hadus]

Hi folks

I've had Proxmox up and running a few months now. Overall, it's been great.
My one issue is whenever a VM has disk activity, the overall IO delay sky rockets. Goes to maybe 70-90%.
This pretty much halts the entire installation, including other VMs and LXC's. But for the life of me I cannot figure out what's wrong...

I ended up defining my storage as mirrors, but I was in massive doubt when doing so.
Have I accidentally configured a peformance killer? Everything is on SSD.

I have attached output of "inxi -Fxz" to show setup and specs.
Extracts below:


Mobo: MSI B350 GAMING PLUS (MS-7A34)
CPU: AMD Ryzen 3 3100

RAID:
Device-1: fast2 type: zfs status: ONLINE level: mirror-0 raw: size: 444 GiB free: 274 GiB
zfs-fs: size: 430.25 GiB free: 260.37 GiB
Components: Online: 1: 2:
Device-2: rpool type: zfs status: ONLINE level: mirror-0 raw: size: 220 GiB free: 65.7 GiB
zfs-fs: size: 213.19 GiB free: 58.92 GiB
Components: Online: 1: nvme0n1p3 2: sdb3
Drives:
Local Storage: total: raw: 1.35 TiB usable: 1.55 TiB used: 101.99 GiB (6.4%)
ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLW256HEHP-000L7 size: 238.47 GiB temp: 51.9 C
ID-2: /dev/sda vendor: SanDisk model: TEAM T253X1480G size: 447.13 GiB
ID-3: /dev/sdb vendor: Kingston model: SH103S3240G size: 223.57 GiB
ID-4: /dev/sdc vendor: Intenso model: SSD SATAIII size: 476.94 GiB


Thanks, Hadus

 

[Falk R.]

You cannot expect good performance with ZFS with these cheap SSDs.

 

[hadus]

Update:

Found another post https://forum.proxmox.com/threads/slow-zfs-performance.51717/
While the solution provided comes at a cost, it has certainly helped. IO Delay is not gone, but the system is now usable during load.
zfs set sync=disabled rpool
IO delay rests at maybe 20-30% with disk activity now.

 

[justinclift]

@hadus To get rid of the performance delay you'd ideally need to switch to some different SSDs (more enterprise oriented).

Does your system have a spare PCIe slot (ie for a SAS card and some cheap SAS SSDs from Ebay), or would you have budget to spring for something like Samsumg PM893's (if you need to stick with SATA)?