New Proxmox install - seeing high IO wait on RAIDZ2 array ?

BenF

New Member
Sep 2, 2019
2
0
1
52
Could do with some setup and configuration advice - particularly as I'm seeing high IO wait times ( 30-70% ) with disk activity on a ZFS RAID-Z2 array.

As background :

Hardware : Supermicro X9 SCL, 16Gb RAM, E3-1220v2, LSI 9220 8i flashed to IT mode. 6 x WD Red 10TB disks, spread with three on the LSI controller, three to on-board SATA controller. Proxmox installed on a separate 160G SSD connected to the on-board SATA. Network connected at 10Gbps via TX540 nic.

Disk setup : All six setup in RAID Z2 array.

I'm running a VM with OpenMedia Vault ( 4GB, 2 cores ) with two Virtual disks ( 10, 15 TB ) passed through and shared over SMB and NFS

The issue I'm seeing :

Writing files NFS to the OpenMediaVault (OMV) VM I get throughput of @ 500-1.2Gbps but I/O wait reported by the Proxmox host is between 30-70% and within the VM pegged at 100%.

Trying to (eg) read a file from OMV at the same time ( eg a 720p video played by VLC ) VLC suffers with frequent playback pauses / stalls - I found these correlate to host IO wait times of over 60%.

I'm concerned that the high I/O wait could impact the performance of the other VMs / applications I am planning to add - eg Zoneminder for CCTV.

Any suggestions as to what I could do to reduce I/O wait times ?

Currently considering :

- Looking at additional memory - however 32GB is the max supported on this motherboard
- Have considered disabling compression on ZFS, but all advice I have found says this should have little / no impact and to leave it enabled
- Startover, and change the array from a six disk RAIDZ2 to a RAID 10 configuration for more IOPs ( although a little less comfortable with risks around this ) ?
- Add a SSD as a ZIL/ARC to a separate channel on the LSI controller?
 
Hi,

16 GB Ram, is not sufficient as zfs will use 8 GB. Also for any storage like OMV is also not sufficient only 4GB.

With so limited resurces, I would only use only a CT with only one service (NFS, or whatever).

How is OMV installed, as a VM or as a CT?
 
Hi,

16 GB Ram, is not sufficient as zfs will use 8 GB. Also for any storage like OMV is also not sufficient only 4GB.

With so limited resurces, I would only use only a CT with only one service (NFS, or whatever).

How is OMV installed, as a VM or as a CT?

Thanks : I've ordered in another 16GB so that will hopefully help, appreciate though that 32GB is low-end with today's hardware - but this is a budget build.

I did consider small Optane 32G NVMe SLOG drive, but doing a lot of reading around I gather that's unlikely to make a big improvement to write performance. Quick tests I've done disabling ZFS logging temporarily didn't show any significant improvement in throughput.

OMV : I set it up as a VM with 2 cores and 4GB RAM ( didn't think to look for a CT ) with 10 and 15TB disks passed through to it. Upping the VM's spec to 8GB RAM and 4 cores didn't make a significant difference to write performance. I didn't think to look for a container version - is that likely to help performance ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!