Crippling performance using zfs on the host

[Dave Siddons]

Our host is a twin xeon with 36 cores, 512TB memory and 6 2TB disks

A default installation of proxmox

The guest OS is Centos 7.6.

The problems first became apparent after seeing a load on the host of 38.20,39.01,38.47

If I issue this command on the Centos guest
dd if=/dev/urandom of=/var/lib/pgsql/test bs=4096 status=progress

It start about 180MB/s and drops slowly to 10MB/s or less and on the host you see load rise from 0.5,0.5,0.5 to the above values.

It prevents access to other guests on this host or, at a minimum, cripples functionality with even ls commands taking an age.

I have some guidance but no real solution to this problem.

Cheers in advance

 

[Dave Siddons]

OK I have more info.

The test is the above dd command

Single node PM server with zfs raidz-1
one centos vm
kick off test in vm and io rate starts at about 200Mb/s
Monitor server load on host and it starts going up pretty much straight away
As it gets to about 20 the I/O rate starts to drop
Server load peaks at 40 and I/O rate continues its decent before I kill it at about 60MB/s
This is bad
If I then delete the test file that has been generated it takes minutes
Server load remains high for some time before slowly returning to normal
Next test
run dame test directly on host
*same
I/O rate peaks at 150MB/s and stays there
Server load got to about 2.5 before killing test
deleting file was instant
VM seems to be going beyond the rate of I/O that the host zfs filesystem can comfortably support
Can this be restricted or controlled in config?

Any help much appreciated

 

[sg90]

What cache setting do you have set on the virtual disk in Proxmox?

 

[Dave Siddons]

Default (no cache)

 

[janssensm]

If your 2TB disks are spinning disks, that could trigger this.
ZFS runs better on non-spinning disks, or a combination of both with ZIL.

If your disks are connected via a hardware raid controller you could be asking for trouble, zfs likes direct access to disks.
So best is to connect via direct ports or HBA (in IT mode)

https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks
https://pve.proxmox.com/wiki/ZFS_on_Linux

 

[Dave Siddons]

Cheers for response

Yes they are spinning disks
No hardware RAID

As we have bundles of memory, would a ramdisk be an alternative to an SSD for Zil and ARC

 

[janssensm]

Maybe, depends if this will be something for testing or production use, you can always try if that works for you.
But rather take a bit of time to read up info about ZFS on linux. There is a lot of info to be found on this forum.
My guess is you already have a nice base system (plenty of memory and cores), a couple of ssd's would fit nice and might save you a headache.

 

[Dave Siddons]

Sadly the servers are full. No more space to add anything and no funding. They are production (or will be). I was hoping a RAM disk would suffice as we can afford to give up 32GB RAM easily.