Continuing testing of disk performance (see previous part at https://forum.proxmox.com/threads/zfs-vm-lxc-disk-performance-benchmarking-part-1-zfs-slow.166701/), so won't repeat the background.
In part 2 I'll focus on tests that seem to show that ZFS vols seem to be unreasonably slow when running under VM.
Test 1 is the same as in the last thread, 10G of 4k writes at iodepth=1 with sync=1:
IOPS:
So in the previous part it turned out that using zvols even on the host drops performance by a factor of 5. At least under LXC performance is not dropping any further (e.g., LXC runs at host speed). But running the same test under VM drops performance by additional factor of ~3! Going from 84k IOPS under LVM on the host to below 7k IOPS zvol in VM is a slowdown by a factor of 12. Ouch.
But wait, there is more (tm)
This time it doesn't seem to be as much about the volumes, but more about virtualization itself. The same test under thick LVM on VM is also very slow - about twice as fast as zvol/VM, but still about 7 times slower than LVM/host. Disabling sync improves speed, but barely.
Now, LVM performance under LXC is commendable at 55k IOPS. Slower than the host, but less than 2x slowdown, and 8 times faster than corresponding VM! (but there will be more on LXC performance in the next part).
Again, did not expect this result. Obviously, some performance penalty was expected, but not THAT much. And it's not that every single test was showing that much difference. For instance, the sequential write test from the follow up in the previous part (max throughput) maxed out CPU on the host reaching 1.6GB/s throughput, and reached about the same numbers under VM. But these low depth queue sync'ed writes look absolutely terrible. Anyone can confirm or disprove on their own data?
KVM hardware virtualization is enabled, AES is enabled. No weird CPU usage on this test.
Bottom line is, some tests show several times slowdown in disk IO comparing to host under VM. Does it sound plausible?
Config data:
In part 2 I'll focus on tests that seem to show that ZFS vols seem to be unreasonably slow when running under VM.
Test 1 is the same as in the last thread, 10G of 4k writes at iodepth=1 with sync=1:
IOPS:
| raw partition, host | 84,000 |
| zvol, host | 18,000 |
| zvol, VM | 6,800 |
| zvol, VM, sync=0 | 10,000 |
| zvol, LXC | 18,000 |
| LVM, host | 84,000 |
| LVM, VM | 12,000 |
| LVM, LXC | 55,000 |
Code:
fio --filename=/dev/zvol/b-zmirror1-nvme01/vm-101-disk-0 --ioengine=libaio --loops=1 --size=10G --time_based --runtime=60 --group_reporting --stonewall --name=cc1 --description="CC1" --rw=write --bs=4k --direct=1 --iodepth=1 --numjobs=1 --sync=1So in the previous part it turned out that using zvols even on the host drops performance by a factor of 5. At least under LXC performance is not dropping any further (e.g., LXC runs at host speed). But running the same test under VM drops performance by additional factor of ~3! Going from 84k IOPS under LVM on the host to below 7k IOPS zvol in VM is a slowdown by a factor of 12. Ouch.
But wait, there is more (tm)
This time it doesn't seem to be as much about the volumes, but more about virtualization itself. The same test under thick LVM on VM is also very slow - about twice as fast as zvol/VM, but still about 7 times slower than LVM/host. Disabling sync improves speed, but barely.Now, LVM performance under LXC is commendable at 55k IOPS. Slower than the host, but less than 2x slowdown, and 8 times faster than corresponding VM! (but there will be more on LXC performance in the next part).
Again, did not expect this result. Obviously, some performance penalty was expected, but not THAT much. And it's not that every single test was showing that much difference. For instance, the sequential write test from the follow up in the previous part (max throughput) maxed out CPU on the host reaching 1.6GB/s throughput, and reached about the same numbers under VM. But these low depth queue sync'ed writes look absolutely terrible. Anyone can confirm or disprove on their own data?
KVM hardware virtualization is enabled, AES is enabled. No weird CPU usage on this test.
Bottom line is, some tests show several times slowdown in disk IO comparing to host under VM. Does it sound plausible?
Config data:
VM uname -a (Debian 12 live):
LXC uname -a (Debian 12):
pveversion -v
ZFS properties:
cat /etc/pve/qemu-server/101.conf
cat /etc/pve/lxc/105.conf
Code:
Linux fiotest 6.1.0-29-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.123-1 (2025-01-02) x86_64 GNU/LinuxLXC uname -a (Debian 12):
Code:
Linux fiopct 6.8.12-10-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-10 (2025-04-18T07:39Z) x86_64 GNU/Linuxpveversion -v
Code:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-10-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8: 6.8.12-10
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph: 19.2.1-pve3
ceph-fuse: 19.2.1-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2ZFS properties:
Code:
Exceeded message limit, but nothing overly exciting here: sync=standard, compression=oncat /etc/pve/qemu-server/101.conf
Code:
agent: 1
boot: order=ide2;net0
cores: 4
cpu: x86-64-v2-AES
ide2: local:iso/debian-live-12.9.0-amd64-standard.iso,media=cdrom,size=1499968K
memory: 2048
meta: creation-qemu=9.0.2,ctime=1740711583
name: fiotest
net0: virtio=BC:24:11:0F:05:39,bridge=vmbr0,firewall=1,tag=11
numa: 0
ostype: l26
scsi0: b-lvm-thk-nvme01:vm-101-disk-0,aio=io_uring,backup=0,cache=none,discard=on,iothread=1,replicate=0,size=11G,ssd=1
scsi1: b-zmirror1-nvme01:vm-101-disk-0,aio=io_uring,backup=0,cache=none,discard=on,iothread=1,replicate=0,size=11G,ssd=1
scsi2: ztest:vm-101-disk-0,aio=io_uring,backup=0,cache=none,discard=on,iothread=1,replicate=0,size=111G,ssd=1
scsi3: b-lvm-thn-nvme01:vm-101-disk-0,aio=io_uring,backup=0,cache=none,discard=on,iothread=1,replicate=0,size=111G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=7e99887f-d252-43b2-9d91-7027cb2f84c8
sockets: 1
vmgenid: 7b4eb2f9-ab24-4264-ac43-d9602341249bcat /etc/pve/lxc/105.conf
Code:
arch: amd64
cores: 4
features: nesting=1
hostname: fiopct
memory: 2048
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:7D:CF:EC,ip=dhcp,tag=11,type=veth
ostype: debian
rootfs: b-zmirror1-nvme01:subvol-105-disk-0,mountoptions=discard;lazytime,size=120G
swap: 0
unprivileged: 1