I want to like VirtIOFS, but...

micush · Apr 9, 2025

It's so slow compared to NFS. Like less than 1/4 the performance in my testing with FIO. I want to use it because it seems like a cleaner implementation on how to share files with the host. But, I cannot take that big of a performance hit. Shame really.

DIRECT:
server:/mnt/server1:# show disk perf . 1G
Physical Server
Testing Storage: /mnt/server1
Testing Size: 1G
Read Performance: 182MB/s IOPS=44.5k
Write Performance: 98.3MB/s IOPS=24.0k

NFS:
server:/mnt/server1:# show disk perf . 1G
QEMU Standard PC (Q35 + ICH9, 2009)
Testing Storage: /mnt/server1
Testing Size: 1G
Read Performance: 101MB/s IOPS=24.6k
Write Performance: 54.3MB/s IOPS=13.3k

VIRTIOFS:
server:/mnt/server11:# show disk perf . 1G
QEMU Standard PC (Q35 + ICH9, 2009)
Testing Storage: /mnt/server11
Testing Size: 1G
Read Performance: 26.4MB/s IOPS=6444
Write Performance: 14.2MB/s IOPS=3474

FIO_COMMANDLINE:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65

xytras · Apr 9, 2025

O

micush said:
It's so slow compared to NFS. Like less than 1/4 the performance in my testing with FIO. I want to use it because it seems like a cleaner implementation on how to share files with the host. But, I cannot take that big of a performance hit. Shame really.

DIRECT:
server:/mnt/server1:# show disk perf . 1G
Physical Server
Testing Storage: /mnt/server1
Testing Size: 1G
Read Performance: 182MB/s IOPS=44.5k
Write Performance: 98.3MB/s IOPS=24.0k

NFS:
server:/mnt/server1:# show disk perf . 1G
QEMU Standard PC (Q35 + ICH9, 2009)
Testing Storage: /mnt/server1
Testing Size: 1G
Read Performance: 101MB/s IOPS=24.6k
Write Performance: 54.3MB/s IOPS=13.3k

VIRTIOFS:
server:/mnt/server11:# show disk perf . 1G
QEMU Standard PC (Q35 + ICH9, 2009)
Testing Storage: /mnt/server11
Testing Size: 1G
Read Performance: 26.4MB/s IOPS=6444
Write Performance: 14.2MB/s IOPS=3474

FIO_COMMANDLINE:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65

Ouch....

rzmeu · Apr 10, 2025

@micush Did you try using cache? I am getting about the same performance as the scsi drives on my VM. But maybe I am using the wrong protocol/cache for VM drives.

Virtiofs:

VM drive(need to investigate more, maybe VirtIO Block si better and cache strategy too):

micush · Apr 10, 2025

Yep. Cache=always is enabled. ZFS draid2 is being used as backing storage.

devilkin · Apr 10, 2025

While my results are not AS dramatic, I also see +- 50% decrease in throughput from NFSv4.

rzmeu · Apr 10, 2025

For me it's acceptable, I've configured everything so it doesn't bring any drawback.
A few drives mounted on the host. Mount Points to an LXC running samba, so I am getting full performance, saturating the 10Gb connection on my local network. Same drives also mounted on a VM with virtiofs, there I have plex, torrent client and other apps using media data on those drives. Been using hookscripts for half a year with no issues.

micush · Apr 10, 2025

Like I said, I **want** to use virtiofs and get rid of NFS for this particular purpose, but the performance hit is just too great for me.

rzmeu · Apr 11, 2025

@micush I am getting almost the same performance as you with "Allow Direct IO" enabled. Do you have it enabled?

micush · Apr 12, 2025

rzmeu said:
@micush I am getting almost the same performance as you with "Allow Direct IO" enabled. Do you have it enabled?

No Direct IO enabled here...

alexskysilk · Apr 12, 2025

I replicated thes OPs test ("Allow Direct IO" enabled.)

tldr: can confirm. On this test, virtiofs delivers 38.7% iops vs NFS. Pretty substantial difference.

to the filesystem mounted via virtiofs:

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=./test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
./test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [m(1)][96.0%][r=38.9MiB/s,w=21.0MiB/s][r=9971,w=5373 IOPS][eta 00m:01s]
./test: (groupid=0, jobs=1): err= 0: pid=18436: Fri Apr 11 20:17:41 2025
  read: IOPS=7212, BW=28.2MiB/s (29.5MB/s)(665MiB/23615msec)
   bw (  KiB/s): min= 3768, max=57656, per=100.00%, avg=29376.35, stdev=12237.29, samples=46
   iops        : min=  942, max=14414, avg=7344.09, stdev=3059.32, samples=46
  write: IOPS=3888, BW=15.2MiB/s (15.9MB/s)(359MiB/23615msec); 0 zone resets
   bw (  KiB/s): min= 1912, max=31096, per=100.00%, avg=15838.26, stdev=6580.58, samples=46
   iops        : min=  478, max= 7774, avg=3959.57, stdev=1645.14, samples=46
  cpu          : usr=1.90%, sys=14.91%, ctx=258076, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=28.2MiB/s (29.5MB/s), 28.2MiB/s-28.2MiB/s (29.5MB/s-29.5MB/s), io=665MiB (698MB), run=23615-23615msec
  WRITE: bw=15.2MiB/s (15.9MB/s), 15.2MiB/s-15.2MiB/s (15.9MB/s-15.9MB/s), io=359MiB (376MB), run=23615-23615msec

to the same filesystem, nfs mounted:

Code:

 fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=./test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
./test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.35
Starting 1 process
./test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=71.9MiB/s,w=39.3MiB/s][r=18.4k,w=10.1k IOPS][eta 00m:00s]
./test: (groupid=0, jobs=1): err= 0: pid=18416: Fri Apr 11 20:16:17 2025
  read: IOPS=18.6k, BW=72.5MiB/s (76.0MB/s)(665MiB/9181msec)
   bw (  KiB/s): min=10264, max=102192, per=99.29%, avg=73678.67, stdev=23963.59, samples=18
   iops        : min= 2566, max=25548, avg=18419.67, stdev=5990.90, samples=18
  write: IOPS=10.0k, BW=39.1MiB/s (41.0MB/s)(359MiB/9181msec); 0 zone resets
   bw (  KiB/s): min= 5696, max=54936, per=99.26%, avg=39713.78, stdev=13003.10, samples=18
   iops        : min= 1424, max=13734, avg=9928.44, stdev=3250.78, samples=18
  cpu          : usr=4.69%, sys=22.36%, ctx=89316, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=72.5MiB/s (76.0MB/s), 72.5MiB/s-72.5MiB/s (76.0MB/s-76.0MB/s), io=665MiB (698MB), run=9181-9181msec
  WRITE: bw=39.1MiB/s (41.0MB/s), 39.1MiB/s-39.1MiB/s (41.0MB/s-41.0MB/s), io=359MiB (376MB), run=9181-9181msec

Ramalama · Apr 12, 2025

I hoped either to get finally rid from NFS, but after reading this, virtiofs seems to be senseless :-(

scyto · Sunday at 05:22

Ramalama said:
I hoped either to get finally rid from NFS, but after reading this, virtiofs seems to be senseless :-(

Depends on the workload - reading this design docs this is designed for lighweight usecases like VMs with containers in them - this is what the mention in one of the slide decks i saw and happens to be EXACTLY the scenario i want it for (to replace glusterFS within my docker host VMs)

rzmeu · Monday at 23:15

alexskysilk said:
I replicated thes OPs test ("Allow Direct IO" enabled.)

But OP said "Allow Direct IO" is not enabled. No idea why he gets such bad performance.

This is what I am getting for a directory on Samsung 970 EVO 500GB mounted with cache=always:

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.36
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=135MiB/s,w=72.3MiB/s][r=34.4k,w=18.5k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1050: Mon Apr 14 20:51:14 2025
  read: IOPS=34.4k, BW=134MiB/s (141MB/s)(665MiB/4958msec)
   bw (  KiB/s): min=133688, max=141688, per=100.00%, avg=137527.11, stdev=2873.66, samples=9
   iops        : min=33422, max=35422, avg=34381.78, stdev=718.42, samples=9
  write: IOPS=18.5k, BW=72.3MiB/s (75.9MB/s)(359MiB/4958msec); 0 zone resets
   bw (  KiB/s): min=70920, max=76168, per=99.96%, avg=74053.33, stdev=1752.99, samples=9
   iops        : min=17730, max=19042, avg=18513.33, stdev=438.25, samples=9
  cpu          : usr=16.93%, sys=45.47%, ctx=168732, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=665MiB (698MB), run=4958-4958msec
  WRITE: bw=72.3MiB/s (75.9MB/s), 72.3MiB/s-72.3MiB/s (75.9MB/s-75.9MB/s), io=359MiB (376MB), run=4958-4958msec

I keep my media files on drives mounted via virtiofs. Qbittorrent, Plex work perfectly. For more speed, I also share the drives to LXC via mount points if needed (to run samba).

alexskysilk · Tuesday at 01:43

rzmeu said:
But OP said "Allow Direct IO" is not enabled.

Not the relevant factor.

rzmeu said:
I keep my media files on drives mounted via virtiofs. Qbittorrent, Plex work perfectly.

Not the relevant factor either

mount the same filesystem using nfs and rerun your benchmark. Neither I nor anyone else is saying virtiofs is not fast ENOUGH for any given application, just that it's observably slower than other methods of connection.

rzmeu · Tuesday at 19:04

alexskysilk said:
mount the same filesystem using nfs and rerun your benchmark.

Did that, getting almost equal results, using the fio command OP provided.

This is what I have for NFS which is configured within LXC with mount point to the same drive:

Code:

/mnt/drive *(rw,async,no_subtree_check,no_root_squash)

This is the config for virtiofs in VM settings:

Code:

cache=always

Host:

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
test: (groupid=0, jobs=1): err= 0: pid=49844: Tue Apr 15 20:03:35 2025
  read: IOPS=130k, BW=509MiB/s (534MB/s)(665MiB/1306msec)
   bw (  KiB/s): min=463232, max=542752, per=96.42%, avg=502992.00, stdev=56229.13, samples=2
   iops        : min=115808, max=135688, avg=125748.00, stdev=14057.28, samples=2
  write: IOPS=70.3k, BW=275MiB/s (288MB/s)(359MiB/1306msec); 0 zone resets
   bw (  KiB/s): min=249176, max=291440, per=96.11%, avg=270308.00, stdev=29885.16, samples=2
   iops        : min=62294, max=72860, avg=67577.00, stdev=7471.29, samples=2
  cpu          : usr=15.10%, sys=68.51%, ctx=63116, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=509MiB/s (534MB/s), 509MiB/s-509MiB/s (534MB/s-534MB/s), io=665MiB (698MB), run=1306-1306msec
  WRITE: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=359MiB (376MB), run=1306-1306msec

Disk stats (read/write):
    dm-1: ios=150963/81293, merge=0/0, ticks=13812/1402, in_queue=15214, util=91.61%, aggrios=170323/91860, aggrmerge=0/0, aggrticks=15789/1533, aggrin_queue=17336, aggrutil=61.93%
  nvme0n1: ios=170323/91860, merge=0/0, ticks=15789/1533, in_queue=17336, util=61.93%

NFS:

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.36
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=148MiB/s,w=80.0MiB/s][r=37.8k,w=20.5k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1302: Tue Apr 15 16:57:21 2025
  read: IOPS=37.6k, BW=147MiB/s (154MB/s)(665MiB/4525msec)
   bw (  KiB/s): min=144040, max=157360, per=100.00%, avg=150659.56, stdev=3966.27, samples=9
   iops        : min=36010, max=39340, avg=37664.89, stdev=991.57, samples=9
  write: IOPS=20.3k, BW=79.3MiB/s (83.1MB/s)(359MiB/4525msec); 0 zone resets
   bw (  KiB/s): min=77904, max=84408, per=100.00%, avg=81229.33, stdev=1903.97, samples=9
   iops        : min=19476, max=21102, avg=20307.33, stdev=475.99, samples=9
  cpu          : usr=12.67%, sys=36.74%, ctx=49948, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=665MiB (698MB), run=4525-4525msec
  WRITE: bw=79.3MiB/s (83.1MB/s), 79.3MiB/s-79.3MiB/s (83.1MB/s-83.1MB/s), io=359MiB (376MB), run=4525-4525msec

Virtiofs:

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test.file --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=65
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.36
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=132MiB/s,w=71.2MiB/s][r=33.8k,w=18.2k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1194: Tue Apr 15 16:53:22 2025
  read: IOPS=37.2k, BW=145MiB/s (152MB/s)(665MiB/4579msec)
   bw (  KiB/s): min=132280, max=167096, per=99.84%, avg=148542.22, stdev=13506.06, samples=9
   iops        : min=33070, max=41774, avg=37135.56, stdev=3376.51, samples=9
  write: IOPS=20.1k, BW=78.3MiB/s (82.1MB/s)(359MiB/4579msec); 0 zone resets
   bw (  KiB/s): min=71456, max=89568, per=99.83%, avg=80081.78, stdev=7516.44, samples=9
   iops        : min=17864, max=22392, avg=20020.44, stdev=1879.11, samples=9
  cpu          : usr=18.09%, sys=45.63%, ctx=186848, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=170317,91827,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=665MiB (698MB), run=4579-4579msec
  WRITE: bw=78.3MiB/s (82.1MB/s), 78.3MiB/s-78.3MiB/s (82.1MB/s-82.1MB/s), io=359MiB (376MB), run=4579-4579msec

alexskysilk · Tuesday at 19:17

oh interesting. gonna retry the test and post results.

scyto · Wednesday at 17:38

alexskysilk said:
Neither I nor anyone else is saying virtiofs is not fast ENOUGH for any given application, just that it's observably slower than other methods of connection.

Indeed, it's also unclear to me if there is anything new here vs the 'won't fix' speed issues filed over multiple years in the virtiofs project itself

I am a little confused about DAX and if / how that is enabled in the proxmox ui? Is it the same as one of the caching modes or the directio setting? it seems DAX only helps in certain workloads

do you have an opinion on what caching mode would be reasonable in a protected system where there is a UPS with managed shutdown (currently I run none)

scyto · Thursday at 19:50

rzmeu said:
cache=always

i don't yet have a mental model for caching and virtioFS, whats your take on the caching modes in terms of write safety vs something like NFS?

rzmeu · Thursday at 22:09

scyto said:
i don't yet have a mental model for caching and virtioFS, whats your take on the caching modes in terms of write safety vs something like NFS?

I am not informed enough to recommend this, I wouldn't use it on data I am not ready to lose. I am still using cache=auto, which I have set up half a year ago with hook scripts. Compared to always, auto loses about 21% performance, using OP's fio command.

If NFS has no issues for you, then I don't see any reasons to migrate as you won't get better performance with virtiofs.

scyto · Thursday at 22:24

@rzmeu thanks that helps

my scenario is a bit different, i have docker swarm VMs that currently have glusterFS installed inside them that i use for bind mounts / persistent data to the docker containers, i have never used NFS for these

I am migrating that to cephFS for a number of 'reasons'

I am starting with VirtioFS to expose the cephFS volume on the proxmox hosts to the docker swarm VMs.
I ahve this working seems to be great with my wordpress container and db running in the docker VM.

This will be compared to:

using a cephFS volume driver (someone is wokring on that at the moment)
installing the cephFS fuse client in the Docker Swarm VM
using the kernel mode cephFS driver

These last two should, i hope, never hit the network and all IO happen in the promox host kernel (i.e. be fast). At some point i will try and have all 3 mapped to the same cephFS FS and test).

thanks for your input, i will leave in on auto, reliability is more important than speed at this point.

I want to like VirtIOFS, but...

Renowned Member

Member

New Member

Renowned Member

Active Member

New Member

Renowned Member

New Member

Renowned Member

Distinguished Member

Renowned Member

Active Member

New Member

Distinguished Member

New Member

Distinguished Member

Active Member

Active Member

New Member

Active Member

We value your privacy