Poor IO preformance

[juliannieborg]

Hi,

I have an iops problem in the VM.

When I test the speed of the ssd with fio with this command:

fio --ioengine=psync --filename=/tmp/test --size=1G --time_based --name=fio --group_reporting --runtime=60 --direct=1 --sync=1 --iodepth=64 --rw=read --bs=4k --numjobs=1

When I mount the disk locally or run this fio command on the disk itself, I have IOPS of around 75k.
Once I boot a vm with the same disk I have max 14k iops. I've tried all cache options and different software and hardware configurations, but I never get above 14k.

Does anyone have an idea what this could be? (and yes I have DC SSD's in my server)

Any advices or opinions are very much appreciated.

Thank you.

 

[Matthias.]

Please share your VM config, qm config $VMID.

 

[juliannieborg]

Hi Matthias,

agent: 1
boot: order=ide2;virtio0;net0
cores: 8
hotplug: disk,network,usb
ide2: none,media=cdrom
memory: 16384
meta: creation-qemu=7.0.0,ctime=1663528823
name: ****
net0: virtio=06:75:7B:F9:33:BA,bridge=vmbr0,firewall=1
net1: virtio=8E:38:24:05:80:43,bridge=vmbr1,firewall=1
net2: virtio=EA:76:A:77:9B:BF,bridge=vmbr10
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=d04ff8a5-f935-46f8-a83b-c9fb9cfa9332
sockets: 2
virtio0: StorageMWH08:vm-627-disk-0,discard=on,iothread=1,size=310G
vmgenid: 728107f1-eb64-435b-a3b3-83276821ec49

I just tried it with a disk with 700k+ iops with the same fio command and even then i always only get 14k iops in the vm

 

[Matthias.]

please use cpu: host and if you're on a multi socket system numa: 1 (you can change that in the GUI)
Which CPU do you have? (type and how many sockets)

What sort of storage is StorageMWH08?

 

[juliannieborg]

The Cpu: host and numa: 1 makes it slightly better (around 18k iops)
I've a dual Xeon E5 2697v2 (so 2 sockets)

StorageMWH08 is a ssd LVM-thin but i've also tried it with a zfs pool and nvme LVM-thin

 

[Matthias.]

Try writethrough as cache.

4k is a small blocksize, try benchmarking with 4M. The disk is virtualized, so you always loose performance. You can PCI passthrough the disks to get rid of this overhead.

Also, on both machines, you might want to benchmark the VM disk directly.

 

[juliannieborg]

Hi @Matthias.

So you say that with any disk no matter if it is ZFS pool or NVME ssd the maximum preformance in a VM is about 20k iops in random read write?

 

[Matthias.]

psync in fio in virtualisation is particularly slow, maybe try different configurations
Virtualisation always has an overhead.
Bad configurations increase that overhead.

 

[spirit]

can you try with --ioengine=libaio ? (or --ioengine=io_uring)

could be interesting to see if it's scale with more jobs --numjobs=X (maybe check the cpu usage of fio process in the vm if it's not using 100% a core)

 

[spirit]

also:
https://www.spinics.net/lists/fio/msg10609.html

"
log_info("note: both iodepth>=1 and synchronous I/O engine are selected, "
+ "queue depth will be capped at 1\n");
"

with iodepth=1, it's quite possible that the virtualisation overhead add latency and reduce iops vs fio on physical server directly.

 

[Chris]

--direct=1 --sync=1 --iodepth=64 --rw=read

With this parameters you only test seq read and iodepth has no effect with psync, not

20k iops in random read write?

Please let me refer you to [0] for some better IOPS or bandwidth test parameter.

Also, try if you see any changes when using virtio-scsi instead of virtio-block.

[0] https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance

 

[spirit]

from my test, with a memory drive, and virtio-scsi, with iodepth=128 , I'm able to reach 200000iops.
Don't have tested with iodepth=1, but I think that indeed it should be something like 15-20kiops with the virtualisation overhead.

It's possible to reduce a little bit the latency, with an drive option "queues" not yet available in the gui, only for virtio-scsi

scsi0: StorageMWH08:vm-627-disk-0,discard=on,iothread=1,queues=8,size=310G

(set the number of queues to number of cpu of the vm).

it should give you around 10-20% less latency.