New Proxmox install - seeing high IO wait on RAIDZ2 array ?

[BenF]

Could do with some setup and configuration advice - particularly as I'm seeing high IO wait times ( 30-70% ) with disk activity on a ZFS RAID-Z2 array.

As background :

Hardware : Supermicro X9 SCL, 16Gb RAM, E3-1220v2, LSI 9220 8i flashed to IT mode. 6 x WD Red 10TB disks, spread with three on the LSI controller, three to on-board SATA controller. Proxmox installed on a separate 160G SSD connected to the on-board SATA. Network connected at 10Gbps via TX540 nic.

Disk setup : All six setup in RAID Z2 array.

I'm running a VM with OpenMedia Vault ( 4GB, 2 cores ) with two Virtual disks ( 10, 15 TB ) passed through and shared over SMB and NFS

The issue I'm seeing :

Writing files NFS to the OpenMediaVault (OMV) VM I get throughput of @ 500-1.2Gbps but I/O wait reported by the Proxmox host is between 30-70% and within the VM pegged at 100%.

Trying to (eg) read a file from OMV at the same time ( eg a 720p video played by VLC ) VLC suffers with frequent playback pauses / stalls - I found these correlate to host IO wait times of over 60%.

I'm concerned that the high I/O wait could impact the performance of the other VMs / applications I am planning to add - eg Zoneminder for CCTV.

Any suggestions as to what I could do to reduce I/O wait times ?

Currently considering :

- Looking at additional memory - however 32GB is the max supported on this motherboard
- Have considered disabling compression on ZFS, but all advice I have found says this should have little / no impact and to leave it enabled
- Startover, and change the array from a six disk RAIDZ2 to a RAID 10 configuration for more IOPs ( although a little less comfortable with risks around this ) ?
- Add a SSD as a ZIL/ARC to a separate channel on the LSI controller?

 

[guletz]

Hi,

16 GB Ram, is not sufficient as zfs will use 8 GB. Also for any storage like OMV is also not sufficient only 4GB.

With so limited resurces, I would only use only a CT with only one service (NFS, or whatever).

How is OMV installed, as a VM or as a CT?

 

[BenF]

Hi,

16 GB Ram, is not sufficient as zfs will use 8 GB. Also for any storage like OMV is also not sufficient only 4GB.

With so limited resurces, I would only use only a CT with only one service (NFS, or whatever).

How is OMV installed, as a VM or as a CT?

Thanks : I've ordered in another 16GB so that will hopefully help, appreciate though that 32GB is low-end with today's hardware - but this is a budget build.

I did consider small Optane 32G NVMe SLOG drive, but doing a lot of reading around I gather that's unlikely to make a big improvement to write performance. Quick tests I've done disabling ZFS logging temporarily didn't show any significant improvement in throughput.

OMV : I set it up as a VM with 2 cores and 4GB RAM ( didn't think to look for a CT ) with 10 and 15TB disks passed through to it. Upping the VM's spec to 8GB RAM and 4 cores didn't make a significant difference to write performance. I didn't think to look for a container version - is that likely to help performance ?

 

[guletz]

Hi,
Any CT is faster conpared with VM that have the same services .

Good luck!