I prepared Proxmox 8 for testing on a Dell R640 with 3 U.2 ZFS drives raidz1-0
I have prepared a virtual machine with Debian 12 VirtIO SCSI single. On proxmox I added zvol and mounted it to mnt. I did comparative tests on Proxmox and VM Debian 12.
There were significant speed differences. Why are there such big differences? Can this be optimized somehow?
My test:
Write:
Proxmox:
# fio --rw=randwrite --name=IOPS-write --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
......
iops : min=28760, max=220698, avg=110632.57, stdev=10203.27, samples=377
.....
WRITE: bw=859MiB/s (901MB/s), 859MiB/s-859MiB/s (901MB/s-901MB/s), io=40.0GiB (42.9GB), run=47673-47673msec
VM Debian 12:
.......
iops : min=47545, max=131939, avg=77228.84, stdev=4438.57, samples=466
......
WRITE: bw=587MiB/s (616MB/s), 587MiB/s-587MiB/s (616MB/s-616MB/s), io=34.4GiB (36.9GB), run=60002-60002msec
Read:
Proxmox:
# fio --rw=read --name=IOPS-read --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
........
iops : min=250794, max=279608, avg=256053.97, stdev=1409.25, samples=161
........
READ: bw=1994MiB/s (2091MB/s), 1994MiB/s-1994MiB/s (2091MB/s-2091MB/s), io=40.0GiB (42.9GB), run=20543-20543msec
VM Debian 12:
..........
iops : min=69568, max=203911, avg=118062.44, stdev=11450.43, samples=355
........
READ: bw=805MiB/s (844MB/s), 805MiB/s-805MiB/s (844MB/s-844MB/s), io=40.0GiB (42.9GB), run=50878-50878msec
Proxmox dd:
# dd if=/dev/random of=test2.img bs=8k count=100000 oflag=dsync
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 77.6469 s, 10.6 MB/s
VM Debian 12 dd:
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 150,607 s, 5,4 MB/s
I have prepared a virtual machine with Debian 12 VirtIO SCSI single. On proxmox I added zvol and mounted it to mnt. I did comparative tests on Proxmox and VM Debian 12.
There were significant speed differences. Why are there such big differences? Can this be optimized somehow?
My test:
Write:
Proxmox:
# fio --rw=randwrite --name=IOPS-write --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
......
iops : min=28760, max=220698, avg=110632.57, stdev=10203.27, samples=377
.....
WRITE: bw=859MiB/s (901MB/s), 859MiB/s-859MiB/s (901MB/s-901MB/s), io=40.0GiB (42.9GB), run=47673-47673msec
VM Debian 12:
.......
iops : min=47545, max=131939, avg=77228.84, stdev=4438.57, samples=466
......
WRITE: bw=587MiB/s (616MB/s), 587MiB/s-587MiB/s (616MB/s-616MB/s), io=34.4GiB (36.9GB), run=60002-60002msec
Read:
Proxmox:
# fio --rw=read --name=IOPS-read --bs=8k --direct=1 --filename=test1.img --numjobs=4 --ioengine=libaio --iodepth=32 --group_reporting --runtime=60 --filesize=10G
........
iops : min=250794, max=279608, avg=256053.97, stdev=1409.25, samples=161
........
READ: bw=1994MiB/s (2091MB/s), 1994MiB/s-1994MiB/s (2091MB/s-2091MB/s), io=40.0GiB (42.9GB), run=20543-20543msec
VM Debian 12:
..........
iops : min=69568, max=203911, avg=118062.44, stdev=11450.43, samples=355
........
READ: bw=805MiB/s (844MB/s), 805MiB/s-805MiB/s (844MB/s-844MB/s), io=40.0GiB (42.9GB), run=50878-50878msec
Proxmox dd:
# dd if=/dev/random of=test2.img bs=8k count=100000 oflag=dsync
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 77.6469 s, 10.6 MB/s
VM Debian 12 dd:
100000+0 records in
100000+0 records out
819200000 bytes (819 MB, 781 MiB) copied, 150,607 s, 5,4 MB/s