[SOLVED] virtiofs and running out of file handles

[kegloadam]

Ran into an edge-case with a PBS virtual machine that had the chunks stored the host, backed by virtiofs. Garbage collection would consistently fail with "ENFILE: File table overflow"

Finding the limit wasn't the easiest since it isn't enforced using the process limits, rather "virtiofsd" has its own internal limit, set by a command line switch, that Proxmox itself doesn't allow editing

My solution was to create a wrapper that replaces virtiofsd with a command line that increases the limit that is sufficient for my use case.
It's probably the best to create a dpkg diversion so the file is preserved when apt upgrades the virtiofs package

dpkg-divert --add --rename --divert /usr/libexec/virtiofsd.real /usr/libexec/virtiofsd

Then, make a shell script that contains the rlimit switch, replacing /usr/libexec/virtiofsd

#!/bin/sh
exec /usr/libexec/virtiofsd.real --rlimit-nofile=10000000 "$@"

Cheers!

 

[dcsapak]

mhmm it could make sense to have that as an option somewhere. If you want you can open a feature request on https://bugzilla.proxmox.com for that

 

[kegloadam]

mhmm it could make sense to have that as an option somewhere. If you want you can open a feature request on https://bugzilla.proxmox.com for that

One could take a good hard look at the command line options of virtiofsd to see what else makes sense in the context of PVE. Exposing --readonly might be a good idea, for example.
Since you suggested it I'll go and open a ticket.

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=7984

 

[pr0j3ctx]

I experienced the same issue with a Linux guest and a VirtIO-FS share containing about 1.3 million files.

In my case, the host filesystem is ext4 on LVM.

virtiofsd continuously increased its number of open file descriptors and hit its limit (~1,000,000 FDs), resulting in:

No more file descriptors available to the guest

followed by:

Too many open files in system

I initially considered increasing --rlimit-nofile, but this only appeared to postpone the problem. The FD count kept growing whenever files were accessed.

After reading Bug 7499, I tested:

--inode-file-handles=prefer

and the issue disappeared immediately. The number of open file descriptors remains low and stable, and memory consumption of virtiofsd is significantly lower.
The Proxmox Team added this option for Windows for a time.

This behavior matches the findings described in Bug 7499, even though my guest is Linux and the backend filesystem is ext4 on LVM rather than ZFS.

Related bug report:

https://bugzilla.proxmox.com/show_bug.cgi?id=7499

Since this directly affects workloads, it would be very helpful if Proxmox could expose both --inode-file-handles and --rlimit-nofile as configurable VirtIO-FS options. This would make it much easier to adapt VirtIO-FS to different workloads and avoid maintaining local workarounds.