FIO Benchmark on the ZVOL. Fast in Proxmox, slow in VM. Same benchmark, same ZVOL

[scar83]

Hello! Title says it all, but here a details:

Host: Proxmox VE 6.3-6
Guest: Ubuntu Server 18.04.2 LTS

SSD: ADATA XPG SX8200 Pro 2 TB (two in ZFS mirror)
ZVOL: 90GB, thin-provised, sync=disabled

Then i benchmarking this ZVOL directly on the Host i get almost maximum performance that this SSD can actually provide
But then i run same fio test inside VM on the very same ZVOL (attached to the VM as data disk and with cache=unsafe !) i get three times worse results

So i dont know what to do... Problem definitely lay in virtualisation layer but where? I even tried both VIRTIO and SCSI connections .

---

Here below benchmarks (fio config and result):

1) On the Host:

fio --name RWRITE --rw=randwrite --filename=/dev/zvol/apool/vm-147-disk-x --size=4g --blocksize=4k --iodepth=1 --numjobs=1 --ioengine=posixaio

IOPS=72.5k, BW=283MiB/s

2) Inside VM:

fio --name RWRITE --rw=randwrite --filename=/dev/sda --size=4g --blocksize=4k --iodepth=1 --numjobs=1 --ioengine=posixaio

IOPS=18.9k, BW=73.9MiB/s

 

[spirit]

you really shouldn't use cache=unsafe, because it's faking fsync, so the benchmark results will never be good.

can you try with cache=none, and use fio with --direct=1

(72.5k of randwrite seem really high for this kind of cheap nvme + zfs.) . Or maybe you are cheating with zfs sync=disable, but you don't want this for production, as you'll have zfs corruption on any power outage)

 

[scar83]

you really shouldn't use cache=unsafe, because it's faking fsync, so the benchmark results will never be good.

can you try with cache=none, and use fio with --direct=1

(72.5k of randwrite seem really high for this kind of cheap nvme + zfs.) . Or maybe you are cheating with zfs sync=disable, but you don't want this for production, as you'll have zfs corruption on any power outage)

I am just trying figure out why VM much slower than actual hardware storage so whats why i turned sync=disable for ZVOL and cache=unsafe for VM. It is like "maximum effort" but it is temporary and only for one lab VM

72.5k of randwrite is somewhat similiar of TweakTown.com results for this ADATA drive: https://www.tweaktown.com/reviews/9...dex.html#Synthetic-Benchmarks:-CDM-and-Anvils

Mine is slightly higher - it is because of mirror i guess... And maybe because they use Windows NTFS and not ZFS

I will try that you suggested and will back with results soon!

 

[guletz]

Or maybe you are cheating with zfs sync=disable, but you don't want this for production, as you'll have zfs corruption on any power outage)

Hi,

Sync disable will not corruption zfs, because any zfs write are done using transactions. By default any data is flush from arc to disk at most 5 sec by default(using transactions).
Even in case of power down, in the worst case you can lose at maximum 5 sec of data from arc, without any corruption on zfs.

---

But if you think that mostly databases use direct io, so in this case, any decent database use a internal jurnal (write the data first on jurnal and then to disks), the risk to lose data is very low in most cases.

can you try with cache=none, and use fio with --direct=1

No, do not try this if you know what you do (except you want to test zfs performance only for a database, but I am not aware what db use 4k sync write)

Also note that zfs is not able to do direct io.

Good luck / Bafta

 

[guletz]

So i dont know what to do... Problem definitely lay in virtualisation layer but where?

Hi,

You do not test what you think ... On host you test how fio write to a file and not to a zvol. Also on VM you teste writing a file on a filesystem w/o is on top on a LVM that is on top on zvol.

Good luck / Bafta !

 

[scar83]

can you try with cache=none, and use fio with --direct=1

I did, so this is results:

1) On the Host

fio --name RWRITE --rw=randwrite --filename=/dev/zvol/apool/vm-147-disk-x --size=4g --direct=1 --blocksize=4k --iodepth=1 --numjobs=1 --ioengine=posixaio

IOPS=27.2k, BW=106MiB/s

2) Inside VM with cache=unsafe

fio --name RWRITE --rw=randwrite --filename=/dev/sda --size=4g --direct=1 --blocksize=4k --iodepth=1 --numjobs=1 --ioengine=posixaio

IOPS=4941, BW=19.3MiB/s

3) Inside VM with cache=none

fio --name RWRITE --rw=randwrite --filename=/dev/sda --size=4g --direct=1 --blocksize=4k --iodepth=1 --numjobs=1 --ioengine=posixaio

IOPS=5084, BW=19.9MiB/s

---

So direct=1 killed performance everywhere (which is expected i guess) and cache=none did not made any difference.

 

[scar83]

Hi,

You do not test what you think ... On host you test how fio write to a file and not to a zvol. Also on VM you teste writing a file on a filesystem w/o is on top on a LVM that is on top on zvol.

Good luck / Bafta !

Hello!

Why, friend? I literally asked fio run test on /dev/zvol/apool/vm-147-disk-x which is ZVOL itself, am i wrong?
Yes, it is a file, but you know how they say - everything in Linux is a file

And no-no, i am assure you, i am not using LVM in this VM at all. I run fio directly on the /dev/sda. Or on the /dev/vdb in case of VIRTIO.
So no LVM, and no filesystem either. Just fio and block-device

I am thinking can it be some undocumented QUEMU safety feature, which is slowing down VM speed?

 

[guletz]

I literally asked fio run test on /dev/zvol/apool/vm-147-disk-x which is ZVOL itself, am i wrong?

No, you are not wrong. But I guess that you do not make such tests without a goal/objective in mind! So the question is what is your goal?

 

[guletz]

So direct=1 killed performance everywhere (which is expected i guess) and cache=none did not made any difference.

Yes is is expected!