I/O into VM almost 10 time slowly then on proxmox host

[nik-weter]

I have 2 SAMSUNG MZQL23T8HCLS-00A07 3.84Tb. Specification says about 180000 IOPS and 4000Mbps writing, 1000000 IOPS and 6800Mbps reading.

I tested disks with fio like that:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=10G --readwrite=randwrite --numjobs=16

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=10G --readwrite=randread --numjobs=16

I tried zfs mirror and mdadm raid1 with lvm. From host pc I got results like that:

W: Jobs: 16 (f=16): [r(16)][8.2%][r=911MiB/s][r=233k IOPS][eta 02m:47s]

R: Jobs: 16 (f=16): [r(16)][21.7%][r=6911MiB/s][r=1769k IOPS][eta 00m:18s]

Its enough for me, pretty fast.

But into VM I see different speed.

W:[r=865MiB/s][r=186k IOPS]

R: [r=902MiB/s][r=240k IOPS]

I used VirtIO SCSI single an IO thread.



Add config vm:

boot: cdn
bootdisk: virtio1
cores: 4
ide2: none,media=cdrom
memory: 6144
name: cloud
net0: virtio=86:26:3A:47:93:56,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=2295c302-8177-4403-8dd1-881f0225ef1f
sockets: 1
tablet: 0
virtio0: data:vm-101-disk-0,iothread=1,size=32G
virtio1: iscsivg:vm-101-disk-0,iothread=1,size=42G

 

[kamilandrz]

I run command:

fio --rw=randrw --name=test --bs=4M --direct=1 --filename=./fio.file --numjobs=8 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G

On bare Ubuntu instance installed on single SSD drive I got a result of read and write around 1.2GB/s. It was rather steady for the whole test run.
On proxmox host with ZFS it was half of Ubuntu's result: ~ 600MB/s
On proxmox guest VM (ubuntu server LTS) it was: half of proxmox result ~ 300-400MB/s. The values were dynamically changing during the test, so seems not too reliable.

I was not expecting such impact of zfs and proxmox.

My hardware:
- 2x SSD Samsung PM9B1 1TB PCIe 4.0 NVMe M.2 2280 MZVL41T0HBLB-00B07
- 64 GB DDR4 RAM
- CPU: AMD Ryzen 5 5600G

 

[kamilandrz]

@nik-weter maybe related to https://github.com/openzfs/zfs/issues/11407#issuecomment-752101544 and https://bugzilla.proxmox.com/show_bug.cgi?id=3224?