[SOLVED] some issues with sata controller passthrough:"Failed to open /dev/vfio/11: No such file or directory"

[nsking]

Good evening, everyone!


I recently tried to switch from an Intel platform to an AMD platform (because AMD's consumer-level platform has better ECC support). However, I encountered some issues when attempting to pass through the SATA controller to a guest VM. After following the relevant documentation and setting up the passthrough, I tried to start the VM but received the following error message:

kvm: -device vfio-pci,host=0000:01:00.2,id=hostpci0.2,bus=pci.0,addr=0x10.2: vfio 0000:01:00.2: failed to open /dev/vfio/11: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

When I checked the binding information under /dev/vfio/, I noticed that there was no VFIO device named "11":

root@pve:~# ll /dev/vfio/
total 0
crw------- 1 root root 510, 1 Sep 24 14:53 10
crw------- 1 root root 510, 0 Sep 24 14:53 9
crw-rw-rw- 1 root root 10, 196 Sep 24 14:51 vfio

In my understanding, that VFIO devices are generated based on IOMMU grouping, and the SATA controller in my system belongs to grouping "10". so, I'm unsure why QEMU tries to bind to a VFIO device named "11" when starting the guest.



My hardware is an ASUS B550M motherboard with an AMD Ryzen 5650GE CPU.


I have already enabled SR-IOV and IOMMU related options in the BIOS, and I've configured the relevant parameters in the boot cmdline:

"root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.13.14-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on kvm_amd.nested=1 pcie_acs_override=downstream,multifunction kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 pci=assign-busses vfio_iommu_type1.allow_unsafe_interrupts=1 acpi_enforce_resources=lax initcall_blacklist=sysfb_init iommu=pt modprobe.blacklist=nouveau vfio-pci.ids=1b21:1062

My VFIO modules are loaded correctly:

root@pve:~# lsmod | grep vfio
vfio_pci 57344 0
vfio_virqfd 16384 1 vfio_pci
irqbypass 16384 2 vfio_pci,kvm
vfio_iommu_type1 40960 0
vfio 36864 2 vfio_iommu_type1,vfio_pci

Here are some other potentially useful pieces of information:

"root@pve:/sys/kernel/iommu_groups# find /sys/kernel/iommu_groups/ -type l | grep "/9/"
/sys/kernel/iommu_groups/9/devices/0000:01:00.0
root@pve:/sys/kernel/iommu_groups# find /sys/kernel/iommu_groups/ -type l | grep "/11/"
/sys/kernel/iommu_groups/11/devices/0000:01:00.2

root@pve:~# dmesg | grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.14-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on kvm_amd.nested=1 pcie_acs_override=downstream,multifunction kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 pci=assign-busses vfio_iommu_type1.allow_unsafe_interrupts=1 acpi_enforce_resources=lax initcall_blacklist=sysfb_init iommu=pt modprobe.blacklist=nouveau vfio-pci.ids=1b21:1062
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.083658] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.13.14-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on kvm_amd.nested=1 pcie_acs_override=downstream,multifunction kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 pci=assign-busses vfio_iommu_type1.allow_unsafe_interrupts=1 acpi_enforce_resources=lax initcall_blacklist=sysfb_init iommu=pt modprobe.blacklist=nouveau vfio-pci.ids=1b21:1062
[    0.378741] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.396163] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.396199] pci 0000:00:01.0: Adding to iommu group 0
[    0.396211] pci 0000:00:02.0: Adding to iommu group 1
[    0.396217] pci 0000:00:02.1: Adding to iommu group 2
[    0.396225] pci 0000:00:02.2: Adding to iommu group 3
[    0.396235] pci 0000:00:08.0: Adding to iommu group 4
[    0.396242] pci 0000:00:08.1: Adding to iommu group 5
[    0.396249] pci 0000:00:08.2: Adding to iommu group 6
[    0.396257] pci 0000:00:14.0: Adding to iommu group 7
[    0.396261] pci 0000:00:14.3: Adding to iommu group 7
[    0.396278] pci 0000:00:18.0: Adding to iommu group 8
[    0.396282] pci 0000:00:18.1: Adding to iommu group 8
[    0.396287] pci 0000:00:18.2: Adding to iommu group 8
[    0.396292] pci 0000:00:18.3: Adding to iommu group 8
[    0.396297] pci 0000:00:18.4: Adding to iommu group 8
[    0.396302] pci 0000:00:18.5: Adding to iommu group 8
[    0.396306] pci 0000:00:18.6: Adding to iommu group 8
[    0.396311] pci 0000:00:18.7: Adding to iommu group 8
[    0.396325] pci 0000:01:00.0: Adding to iommu group 9
[    0.396334] pci 0000:01:00.1: Adding to iommu group 10
[    0.396342] pci 0000:01:00.2: Adding to iommu group 11
[    0.396350] pci 0000:02:00.0: Adding to iommu group 12
[    0.396358] pci 0000:02:01.0: Adding to iommu group 13
[    0.396365] pci 0000:02:04.0: Adding to iommu group 14
[    0.396377] pci 0000:03:00.0: Adding to iommu group 15
[    0.396396] pci 0000:04:00.0: Adding to iommu group 16
[    0.396415] pci 0000:04:02.0: Adding to iommu group 17
[    0.396435] pci 0000:04:06.0: Adding to iommu group 18
[    0.396455] pci 0000:04:0e.0: Adding to iommu group 19
[    0.396483] pci 0000:05:00.0: Adding to iommu group 20
[    0.396511] pci 0000:06:00.0: Adding to iommu group 21
[    0.396522] pci 0000:09:00.0: Adding to iommu group 22
[    0.396528] pci 0000:0b:00.0: Adding to iommu group 23
[    0.396542] pci 0000:0c:00.0: Adding to iommu group 24
[    0.396550] pci 0000:0c:00.1: Adding to iommu group 25
[    0.396559] pci 0000:0c:00.2: Adding to iommu group 26
[    0.396568] pci 0000:0c:00.3: Adding to iommu group 27
[    0.396576] pci 0000:0c:00.4: Adding to iommu group 28
[    0.396585] pci 0000:0c:00.6: Adding to iommu group 29
[    0.396592] pci 0000:0d:00.0: Adding to iommu group 30
[    0.396889] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.434332] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.395220] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

I work as a Linux administrator, but I have limited knowledge in the field of virtualization. I would greatly appreciate everyone's help and support, as I'm seeking assistance in this area. Thank you in advance for any help you can provide.

 

[nsking]

I tried changing the motherboard to an ASUS TUF GAMING B550 PLUS, but it looks like the problem persists.

 

[leesteken]

There are several threads on the forum about passthrough of SATA controllers and those devices not resetting properly (and therefore not working in a VM), especially on-board ones.
Since you are using pcie_acs_override=downstream,multifunction the IOMMU group information is "fake" and the SATA controller is probably part of the big "chipset group" unless you use a X570(S) motherboard.
The usual advise for passthrough is: search for a success story and buy the exact same hardware. Maybe you can find a SATA controller that is known to work with passthrough and put in the first x16 PCIe slot (because other slots are part of the big "chipset group")?

 

[nsking]

There are several threads on the forum about passthrough of SATA controllers and those devices not resetting properly (and therefore not working in a VM), especially on-board ones.
Since you are using pcie_acs_override=downstream,multifunction the IOMMU group information is "fake" and the SATA controller is probably part of the big "chipset group" unless you use a X570(S) motherboard.
The usual advise for passthrough is: search for a success story and buy the exact same hardware. Maybe you can find a SATA controller that is known to work with passthrough and put in the first x16 PCIe slot (because other slots are part of the big "chipset group")?

Thanks for your reply, however I've found a solution to the problem: don't select "All Functions" when adding the pcie device. I don't know why, but checking "All Functions" causes qemu to try to use the wrong vfio device. Now all the issues have been resolved and my VM is running fine.