Proxmox VFIO [SOLVED]

[Giggling3999]

How does Proxmox (8.1.10) set VFIO automatically?

I have a VM, with passed through PCIe NVME. I am having issues with speeds on this, so wanted to test the zpool on the host machine. All devices SEEM to be passing through fine

I am new to proxmox, but previously used libvirt.

I have set vfio to be loaded at boot, as per instructions, and I can see now I shut down my VM - That lspci tells me a vfio driver was loaded for my NVME.

What I want to do (temporarily) is load the nvme driver for these, then test, then return them to the VM.
The problem is I never specified directly for the vfio driver to be loaded for these devices, so I assume there is some automagic happening in the background to set it for devices that are passed through?

 

[Giggling3999]

For example; I don't understand how vfio was set automatically for this nvme

0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
        Kernel driver in use: vfio-pci
        Kernel modules: nvme

But not this one

04:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)
        Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD
        Kernel driver in use: nvme
        Kernel modules: nvme

As I never set vfio manually with the device hex (e.g. vfio: 1022:149c) like I would have with my previous non Proxmox setups

 

[dcsapak]

for passthrough to work at all the device must be bound to the vfio-pci driver, so we try to do that on vm start
but we don't load the old driver for these devices on shutdown, since that sometimes does not work so well (depending on the driver), we'd have to save the original module somewhere for each device, which is not useful for most people as they don't want to use them alternating between host and guest

i guess you could use a vm hookscript to load the original module for the device in the post stop hook step

 

[Giggling3999]

Thanks for the reply; that makes sense now

For my use case I am doing it only the once for testing purposes (I think I am having hardware issues to be honest)

So in theory; if I disable autostart on the VM, and reboot proxmox, VFIO would not be loaded.

To be cheeky - Is there a command I can use to do with manually (So I wouldn't have to restart proxmox)

 

[dcsapak]

To be cheeky - Is there a command I can use to do with manually (So I wouldn't have to restart proxmox)

thats just a kernel feature to bind/unbind drivers from a device

see e.g. here: https://lwn.net/Articles/143397/

we basically do the same thing for the vfio-pci driver, we first 'unbind' the old driver, and then 'bind' the device to 'vfio-pci'

 

[Giggling3999]

Thanks so much for this

I will have a go at this later when I have more brainpower!

 

[Giggling3999]

For anyone wanting to do this in future; it was actually very easy - At least for my NVME drives

Stop my VM;

tree /sys/bus/pci/drivers
echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/nvme/bind
echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/nvme/bind

Do locally what I wanted with the nvme driver

tree /sys/bus/pci/drivers 
echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/nvme/unbind 
echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/nvme/unbind 
echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

Restart the VM - No problems