Big difference in speed tests on the proxmox host and on the virtual machine

[tomeks11]

I prepared Proxmox 8 for testing on a Dell R640 with 3 U.2 ZFS drives raidz1-0

I have prepared a virtual machine with Debian 12 VirtIO SCSI single. On proxmox I added zvol and mounted it to mnt. I did comparative tests on Proxmox and VM Debian 12.
There were significant speed differences. Why are there such big differences? Can this be optimized somehow?

My test:

Write:

Proxmox:
# fio --rw=randwrite --name=IOPS-write --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
......
iops : min=28760, max=220698, avg=110632.57, stdev=10203.27, samples=377
.....
WRITE: bw=859MiB/s (901MB/s), 859MiB/s-859MiB/s (901MB/s-901MB/s), io=40.0GiB (42.9GB), run=47673-47673msec

VM Debian 12:
.......
iops : min=47545, max=131939, avg=77228.84, stdev=4438.57, samples=466
......
WRITE: bw=587MiB/s (616MB/s), 587MiB/s-587MiB/s (616MB/s-616MB/s), io=34.4GiB (36.9GB), run=60002-60002msec

Read:

Proxmox:
# fio --rw=read --name=IOPS-read --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
........
iops : min=250794, max=279608, avg=256053.97, stdev=1409.25, samples=161
........
READ: bw=1994MiB/s (2091MB/s), 1994MiB/s-1994MiB/s (2091MB/s-2091MB/s), io=40.0GiB (42.9GB), run=20543-20543msec

VM Debian 12:
..........
iops : min=69568, max=203911, avg=118062.44, stdev=11450.43, samples=355
........
READ: bw=805MiB/s (844MB/s), 805MiB/s-805MiB/s (844MB/s-844MB/s), io=40.0GiB (42.9GB), run=50878-50878msec


Proxmox dd:
# dd if=/dev/random of=test2.img bs=8k count=100000 oflag=dsync
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 77.6469 s, 10.6 MB/s

VM Debian 12 dd:
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 150,607 s, 5,4 MB/s

 

[leesteken]

raidz1 has a lot of padding, and especially when your volblocksize of the virtual disk is not optimal, this amplifies reads and writes, which lowers performance. I don't know enough of the details to explain it better. There are quite a lot of threads about ZFS raidz1 on this forum, but mostly about unexpected little space on raidz1.

 

[tomeks11]

But why is there such a difference between tests on Proxmox with raidz1 and VM ext4. The VM also uses ZFS with raidz1

 

[leesteken]

But why is there such a difference between tests on Proxmox with raidz1 and VM ext4. The VM also uses ZFS with raidz1

Please share the VM configuration file (qm config number), so we can see how the virtual disk inside the VM is defined.

 

[tomeks11]

agent: 1
boot: order=scsi0;net0
cores: 16
cpu: host
machine: q35
memory: 65536
meta: creation-qemu=8.0.2,ctime=1698310399
name: k-firebird-superclassic
net0: virtio=9E:54:22:17:52:99,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: data-zfs:vm-101-disk-0,iothread=1,size=32G,ssd=1
scsi1: data-zfs:vm-101-disk-1,iothread=1,size=200G,ssd=1
scsi2: data-zfs:vm-101-disk-2,iothread=1,size=250G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=229d80fb-347e-4066-bac2-82434dc57775
sockets: 2
vmgenid: ebfca7ed-4f2c-45b1-abd6-268a8c487233

 

[leesteken]

scsi0: data-zfs:vm-101-disk-0,iothread=1,size=32G,ssd=1
scsi1: data-zfs:vm-101-disk-1,iothread=1,size=200G,ssd=1
scsi2: data-zfs:vm-101-disk-2,iothread=1,size=250G,ssd=1
scsihw: virtio-scsi-single

There is overhead in virtual disks. Ext4 uses 4k blocks, while QEMU shows 512 bytes sectors, but ZFS uses some volblocksize (check with zfs get volblocksize) and that causes amplification. Then there is also your 3-drive raidz1, which adds padding and more amplification. All the extra bytes the need to be read or written will decrease performance.

 

[tomeks11]

So what is the possibility of using the storage power from ZFS in a VM?
When I tested LVM in proxmox and in VM, the differences were much smaller.