Virtiofs - high usage of cached and shared memory

P

purelinux

New Member
May 1, 2025
2
0
1
Hello together,
a few weeks ago, i configured a VirtioFS for one of my VMs on one of my Proxmox Hosts. So far, its working pretty well - good performance and high stability. Its a great feature, which i am really thankful for. Thanks, Proxmox Team!

About my setup: Proxmox Host with 128 GB regECC memory and NVMe as well as HDD based ZFS pools (only ZFS, no other kind of storage). As a result of that, all caching of the filesystem is basically done by ZFS ARC.

I configured the VirtioFS share like this:

/etc/pve/mapping/directory.cfg
Code: 
shared-data
    map node=my-proxmox-host,path=/hdd-mirror-1/data/shared/
    description Shared data

/etc/pve/qemu-server/183.conf
Code: 
...
virtiofs0: shared-data,cache=never,expose-acl=1
...

(in the VM 183) /etc/fstab
Code: 
...
shared-data                           /app/shared-data virtiofs ro,relatime 0 0
...

So imho a normal setup with disabled cache, as the caching is already done by ZFS ARC.

But then, i saw this in my monitoring:
1746104453792.png

Yes, the point where the usage of the huge amount of Cached and Shared started, was exactly the moment, when i enabled the VirtioFS. A look into htop proofs it - more than 49 GiB of virtual memory.
1746104660654.png

Of course, this kind of memory should be normally available for other processes, if explicitly required, but imho its still missing for the ZFS ARC. And its still not really comprehensible for me, what VirtioFS is doing with that much memory with disabled caching.

Just for the protocol: I also tried the other options for cache, which are available with VirtioFS, but this doesnt changed anything. It feels a bit like a dead option, which is doing nothing.

Does anybody know, why VirtioFS uses that much buffers/shared memory? Or, is anybody experiencing the same behavior on his/her Proxmox box?
 
I had some time for more research.

VMs are started with a few more parameters, if virtiofs is enabled:
  • -object memory-backend-memfd,id=virtiofs-mem,size=49152M,share=on
  • -chardev socket,id=virtiofs0,path=/run/qemu-server/virtiofsd/vm183-fs0
  • -device vhost-user-fs-pci,queue-size=1024,chardev=virtiofs0,tag=shared-data
  • -machine memory-backend=virtiofs-mem,type=pc+pve1
The first of these parameters is important, as it seems like the VM is started with a memory-backend-memfd, which is as large as the configured memory of the VM (48 GiB). This is also making sense, as cache and shared memory together results in ~48 GiB.

I tried to hardcode the parameters value to a lower value (1G) for testing, but then the `qm start 183` command resulted in

kvm: Machine memory size does not match the size of the memory backend
start failed: QEMU exited with code 1

So it looks like this is a hard requirement of virtiofsd - the official docs are stating something similar.

So the obvious "solution" seems to be lowering the memory, which is provisioned to the VM. But thats in fact not a solution, as the memory is required by the processes in the VM.

I am wondering, what this memory-backend-memfd object is doing. Does anybody know?

`lsof` is showing these memory-backend-memfd objects, but it shows them as deleted:
Code: 
# lsof -s |head -n1 && lsof -s | grep -F "memfd:memory-backend-memfd"
COMMAND       PID     TID TASKCMD               USER   FD      TYPE             DEVICE        SIZE       NODE NAME
virtiofsd 3406761                               root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
virtiofsd 3406761                               root   15u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761                               root   18u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761                               root   19u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406762 vring_wor             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
virtiofsd 3406761 3406762 vring_wor             root   15u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406762 vring_wor             root   18u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406762 vring_wor             root   19u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406774 virtiofsd             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
virtiofsd 3406761 3406774 virtiofsd             root   15u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406774 virtiofsd             root   18u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
virtiofsd 3406761 3406774 virtiofsd             root   19u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766                               root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766                               root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406767 call_rcu              root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406767 call_rcu              root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406768 kvm                   root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406768 kvm                   root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406769 kvm                   root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406769 kvm                   root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406770 kvm                   root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406770 kvm                   root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406771 kvm                   root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406771 kvm                   root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406803 vhost-340             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406804 CPU\x200/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406804 CPU\x200/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406805 CPU\x201/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406805 CPU\x201/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406806 CPU\x202/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406806 CPU\x202/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406807 CPU\x203/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406807 CPU\x203/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406808 CPU\x204/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406808 CPU\x204/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406809 CPU\x205/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406809 CPU\x205/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406810 CPU\x206/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406810 CPU\x206/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406811 CPU\x207/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406811 CPU\x207/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406812 CPU\x208/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406812 CPU\x208/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406813 CPU\x209/             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406813 CPU\x209/             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406814 CPU\x2010             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406814 CPU\x2010             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406815 CPU\x2011             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406815 CPU\x2011             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406816 CPU\x2012             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406816 CPU\x2012             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406817 CPU\x2013             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406817 CPU\x2013             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406818 CPU\x2014             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406818 CPU\x2014             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406819 CPU\x2015             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406819 CPU\x2015             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406820 CPU\x2016             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406820 CPU\x2016             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406821 CPU\x2017             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406821 CPU\x2017             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406822 CPU\x2018             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406822 CPU\x2018             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406823 CPU\x2019             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406823 CPU\x2019             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406824 CPU\x2020             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406824 CPU\x2020             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406825 CPU\x2021             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406825 CPU\x2021             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406826 CPU\x2022             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406826 CPU\x2022             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406827 CPU\x2023             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406827 CPU\x2023             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406828 CPU\x2024             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406828 CPU\x2024             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406829 CPU\x2025             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406829 CPU\x2025             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406830 CPU\x2026             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406830 CPU\x2026             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406831 CPU\x2027             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406831 CPU\x2027             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406834 vnc_worke             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3406834 vnc_worke             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3406835 kvm-nx-lp             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3407090 iou-wrk-3             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3407090 iou-wrk-3             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3408466 iou-wrk-3             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3408466 iou-wrk-3             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3413935 iou-wrk-3             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3413935 iou-wrk-3             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)
kvm       3406766 3417236 iou-wrk-3             root  DEL       REG                0,1                   4108 /memfd:memory-backend-memfd
kvm       3406766 3417236 iou-wrk-3             root   37u      REG                0,1 51539607552       4108 /memfd:memory-backend-memfd (deleted)

I don't really see through that all... seems like i need to dig a little bit deeper.
 
Last edited:
You must log in or register to reply here.
Top Bottom