virtiofsd is used to "passthrough" parts of a host filesystem to a virtual machine with "local filesystem symantics and performance". Proxmox moved to a rust based version of virtiofsd in PVE 8 located here. It's installed as a separate package called "virtiofsd" in PVE 8, whereas a different, non-rust based version of virtiofsd, was "included" in PVE 7.x. Note there are several front-ends and back-ends that can be used. What is documented below works for me. Make sure you test well before putting anything into "production".
As was pointed out recently (thank you @fabian ) a future PVE 8.x version will have WebUI support for virtiofsd (see here for the patch). I fully expect these instructions will have a limited "shelf life" but until then, below is what I did to implement the new virtiofsd in PVE 8.0.x in four areas:
Substitute "changeme.sock", "changeme_tag" and "size" above to appropriate values where:
changeme.sock=full path name of the socket file used for IPC between VM and host
changem_tag=name used inside the VM for the mount command, or in /etc/fstab
size=size of RAM used by the VM (only tested with it equal in size to VM ram)
NOTE: once an "args:" line has been added, the VM will not start unless the socket file exists. virtiofsd is what creates the socket file (see next step).
Confirm args: is a part of the configuration with the command:
And to "see" the way KVM/Qemu runs the VM:
Note the last 3 or 4 lines should contain the components of the args: line.
Substitue:
changeme.sock with the name used in args: in step (1)
/dir/to/share/with/vm with the underlying host directory exposed to the VM
Instead of "--syslog", use "--log-level debug" to troubleshoot. For additional/different options, see the virtiofsd documentation REAME file here.
To manually start it as a daemon, preface it with nohup and background it with ampersand (&) like:
Special NOTE, this version of virtiofsd has to be "double-forked" to run as a damon. This means once started, there will be two virtiofsd processes, a "parent" and "child". If you examine the PIDs, the "grandparent" PID has to be 1. The "grandparent" must be PID 1. More on this in the next section.
Once running, start the VM as you normally would and see how to mount the shared directory in Linux in step (4).
This will automatically start virtiofsd before the VM.
If needed, see Hookscripts in the PVE documentation here.
Of the four phases of a hookscript, virtiofsd needs to run in the pre-start phase.
If your hookscripts are written in perl, I can not comment how to start virtiofsd since all of my hookscripts are in bash.
(Please feel free to add a perl equivalent in the comments.)
If your hookscripts are written in Bash, starting virtiofsd can be problematic since it needs to be double-forked. For complete coverage of what that means see here. After trying many commands including: Bash here documents, sourcing that here doc, eval(), setsig(1), nohup and a few others, the way I found that worked was to create a one-time systemd service unit with systemd-run(1).
In the "pre-start" phase of your bash hookscript, start virtiofsd with:
Substitute:
changeme_service with a systemd service name of your choice
changeme.sock with the name used in the args: line in step (1)
/dir/to/share/with/vm for the underlying host directory exposed to the VM
Afterwards, the systemd service unit can then be examined with:
Then issue the following mount using the changeme_tag defined in step 1:
Test with:
If something is wrong, the command will appear to "hang".
Once working, and with a working hookscript, add the mount to the /etc/fstab permanently, like the following:
See fstab(5) for other options if desired.
Laslty, vitiofsd will shut down automatically after the VM that is using it stops. There is no need to stop it from the hookscript. This is by design.
As was pointed out recently (thank you @fabian ) a future PVE 8.x version will have WebUI support for virtiofsd (see here for the patch). I fully expect these instructions will have a limited "shelf life" but until then, below is what I did to implement the new virtiofsd in PVE 8.0.x in four areas:
- <VMID>.conf changes
- virtiofsd on the host
- hookscript additions
- Mount inside the Linux VM
Directions
1. Add an "args:" line to the VM's VMID.conf file located in /etc/pve/qemu-server.
Please note the documentation here states this is for experts only.
Bash:
args: -chardev socket,id=virtfs0,path=/run/changme.sock -device vhost-user-fs-pci,queue-size=1024,chardev=virtfs0,tag=changeme_tag -object memory-backend-file,id=mem,size=32768M,mem-path=/dev/shm,share=on -numa node,memdev=memSubstitute "changeme.sock", "changeme_tag" and "size" above to appropriate values where:
changeme.sock=full path name of the socket file used for IPC between VM and host
changem_tag=name used inside the VM for the mount command, or in /etc/fstab
size=size of RAM used by the VM (only tested with it equal in size to VM ram)
NOTE: once an "args:" line has been added, the VM will not start unless the socket file exists. virtiofsd is what creates the socket file (see next step).
Confirm args: is a part of the configuration with the command:
qm config VMID And to "see" the way KVM/Qemu runs the VM:
qm showcmd VMID --pretty Note the last 3 or 4 lines should contain the components of the args: line.
2. Start virtiofsd on the host.
To test, virtiofsd can be started manually, but needs to be started before the VM so it can create the socket *.sock file referenced in "args:". To start manually, use: /usr/libexec/virtiofsd --syslog --socket-path /run/changeme.sock --shared-dir /dir/to/share/with/vm --announce-submounts --inode-file-handles=mandatory Substitue:
changeme.sock with the name used in args: in step (1)
/dir/to/share/with/vm with the underlying host directory exposed to the VM
Instead of "--syslog", use "--log-level debug" to troubleshoot. For additional/different options, see the virtiofsd documentation REAME file here.
To manually start it as a daemon, preface it with nohup and background it with ampersand (&) like:
nohup command_above &Special NOTE, this version of virtiofsd has to be "double-forked" to run as a damon. This means once started, there will be two virtiofsd processes, a "parent" and "child". If you examine the PIDs, the "grandparent" PID has to be 1. The "grandparent" must be PID 1. More on this in the next section.
Once running, start the VM as you normally would and see how to mount the shared directory in Linux in step (4).
3. Start virtiofsd in a hookscript
This will automatically start virtiofsd before the VM.
If needed, see Hookscripts in the PVE documentation here.
Of the four phases of a hookscript, virtiofsd needs to run in the pre-start phase.
If your hookscripts are written in perl, I can not comment how to start virtiofsd since all of my hookscripts are in bash.
(Please feel free to add a perl equivalent in the comments.)
If your hookscripts are written in Bash, starting virtiofsd can be problematic since it needs to be double-forked. For complete coverage of what that means see here. After trying many commands including: Bash here documents, sourcing that here doc, eval(), setsig(1), nohup and a few others, the way I found that worked was to create a one-time systemd service unit with systemd-run(1).
In the "pre-start" phase of your bash hookscript, start virtiofsd with:
Bash:
systemd-run --unit=changeme_service /usr/libexec/virtiofsd --syslog --socket-path /run/changeme.sock --shared-dir /dir/to/share/with/vm --announce-submounts --inode-file-handles=mandatorySubstitute:
changeme_service with a systemd service name of your choice
changeme.sock with the name used in the args: line in step (1)
/dir/to/share/with/vm for the underlying host directory exposed to the VM
Afterwards, the systemd service unit can then be examined with:
systemctl status changeme_service # or the name you changed it to4. Mount it inside the Linux VM
To mount it manually, make (or determine) a directory to use as a mount point:mkdir /mnt/mountpointThen issue the following mount using the changeme_tag defined in step 1:
mount -t virtiofs changeme_tag /mnt/mountpoint Test with:
ls -l /mnt/mountpointIf something is wrong, the command will appear to "hang".
Once working, and with a working hookscript, add the mount to the /etc/fstab permanently, like the following:
Code:
changeme_tag /mnt/mountpoint virtiofs defaultsSee fstab(5) for other options if desired.
Laslty, vitiofsd will shut down automatically after the VM that is using it stops. There is no need to stop it from the hookscript. This is by design.
