Virtiofs reads are faster than the host

[proxmox-fan]

TL; DR: Using dd I'm getting ~550 MB/s reads in a VM guest on virtiofs, but only ~150 MB/s running dd on the host

Here's an odd one, I have a Windows OS template that I developed in a PVE VM that will be V2P into a physical box for testing. I gracefully shut down the guest Windows OS, took a snapshot, then updated the VM config with a virtiofs passthrough and an Ubuntu 24.04 live ISO on the DVD drive.

I booted into the live ISO, successfully installed virtiofsd and clonezilla, and captured a full disk image to the mounted virtiofs path without any issues.

# mount -t virtiofs $passthrough-name /mnt/virtiofs

Out of curiosity, I ran dd against the captured image file to see how well virtiofs performed with reads, and observed ~550 MB/s:

# dd if=sda3.ntfs-ptcl-img.zst.aa of=/dev/null -status=progress
3874079232 bytes (3.9 GB, 3.6 GiB) copied, 7.00021 s, 553 MB/s
^C

To account for Guest OS filesystem caching to memory, I completely unmounted and remounted the virtiofs assthrough and received the same results.

I perform the same read on the host, and it hits a ceiling of ~150 MB/s:
# dd if=sda3.ntfs-ptcl-img.zst.aa of=/dev/null -status=progress
701627392 bytes (702 MB, 669 MiB) copied, 4.53475 s, 155 MB/s
^C

Any idea why the VM guest has faster reads?

Host: Proxmox 8.4.6
Storage: 2TB AData SX8200 NVMe SSD in single-disk ZFS layered over a LUKS encrypted block
The stored clonezilla image files are located in a folder within the ZFS dataset rpool/data/enc

# lsblk -fT -o NAME,FSTYPE,FSVER,LABEL,FSAVAIL,FSUSE%,SIZE,TYPE,MOUNTPOINTS /dev/nvme0n1
NAME            FSTYPE      FSVER    LABEL FSAVAIL FSUSE%  SIZE TYPE  MOUNTPOINTS
nvme0n1                                                    1.9T disk 
├─nvme0n1p1                                               1007K part 
├─nvme0n1p2     vfat        FAT32           510.7M     0%  512M part  /boot/efi
├─nvme0n1p3     LVM2_member LVM2 001                        31G part 
│ └─pve-root    ext4        1.0              15.9G    42%   31G lvm   /
└─nvme0n1p4     crypto_LUKS 2                              1.8T part 
  └─nvme0n1-enc zfs_member  5000     rpool                 1.8T crypt

# zpool list rpool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  1.83T   759G  1.09T        -         -    15%    40%  1.00x    ONLINE  -

# zpool status -v rpool
  pool: rpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:10:28 with 0 errors on Sun Oct 12 00:34:29 2025
config:

        NAME                                                                STATE     READ WRITE CKSUM
        rpool                                                               ONLINE       0     0     0
          dm-uuid-CRYPT-LUKS2-b30f5771759d4ca6972a5da56f7eef01-nvme0n1-enc  ONLINE       0     0     0

errors: No known data errors

# zfs list -o space,sync,xattr,atime,relatime,refquota rpool/enc/data
NAME            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  SYNC      XATTR  ATIME  RELATIME  REFQUOTA
rpool/enc/data  1.03T   111G        0B   98.8G             0B      12.2G  standard  sa     off    on            none

 

[Johannes S]

Since for VirtIO FS to work the filesystem already needs to be mounten on the host I doubt that Passthrough is relevant here. My guess is caching. @OP: Are you sure the data wasn't accessed before the test on the vm?