PCI NIC passthrough issues.

P

patb

New Member
Apr 15, 2022
7
2
3
I hope I can provide the info required to help. I'm a little lost with this whole passthrough thing, most of my VM experience has been cloud based. Thanks in advance. I'm here to help you help me.

Purpose - Passthrough 2nd NIC PCI card to pfsense vm.
Card: Jeirdus Intel Chipset 82546 Dual Port 8492MT PCI Server Network Card RJ45 NIC

GUI TASK ERROR
Task viewer: VM 102 - Start OutputStatus Stop kvm: -device vfio-pci,host=0000:04:01.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on: vfio 0000:04:01.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy TASK ERROR: start failed: QEMU exited with code 1

DMESG ERROR
[ 2347.994353] vfio-pci 0000:04:01.0: enabling device (0000 -> 0003) [ 2348.032000] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:04:01.0)) vs. 00000080 (ehci_hcd:usb2)

/etc/network/interfaces content
auto lo iface lo inet loopback allow-hotplug eno1 iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.0.203/24 gateway 192.168.0.1 bridge-ports eno1 bridge-stp off bridge-fd 0

CHANGES TO /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

CONTENTS OF /etc/modules
vfio vfio_iommu_type1 vfio_pci vfio_virqfd

CONTENTS OF /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

CONTENTS OF /etc/modprobe.d/pve-blacklist.conf
blacklist nvidiafb
 
Can you please show us the contents of the VM configuration file (102.conf in the /etc/pve/qemu-server/ directory)? And can you show us the IOMMU groups using this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done?
 
LEESTEKEN! Thanks for your response! I've attached the results to the commands you outlined below. Thanks for your help. Cheers.

102.conf contents
boot: order=virtio0;scsi2 cores: 2 hostpci0: 0000:04:01 hostpci1: 0000:04:01 memory: 2048 meta: creation-qemu=6.1.0,ctime=1650039566 name: pfsense numa: 0 ostype: l26 scsi2: local:iso/pfSense-CE-2.5.1-RELEASE-amd64.iso,media=cdrom,size=632012K scsihw: virtio-scsi-pci smbios1: uuid=aa763a89-8b80-4791-a67a-9e0dc883ef18 sockets: 1 virtio0: local-lvm:vm-102-disk-0,size=32G vmgenid: 06f3e780-8629-40a0-9445-70cb92f10016

IOMMU GROUPS
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) IOMMU group 10 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0) IOMMU group 11 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] IOMMU group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4] IOMMU group 12 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] IOMMU group 12 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2] IOMMU group 13 05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242] IOMMU group 14 06:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01) IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) IOMMU group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) IOMMU group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1) IOMMU group 2 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] IOMMU group 3 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba] IOMMU group 4 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] IOMMU group 5 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] IOMMU group 6 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0] IOMMU group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) IOMMU group 8 00:1c.3 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev d0) IOMMU group 8 03:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04) IOMMU group 8 04:01.0 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) IOMMU group 8 04:01.1 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) IOMMU group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)
 
This cannot work:
hostpci0: 0000:04:01 hostpci1: 0000:04:01
Probably you selected 04:01.0 and 04:01.1 and selected All Functions for both. But both are functions (sub-devices) of 04:01 and now Proxmox tries to passthrough the same device (with all functions) twice. Just remove the PCI Device (hostpci1) from the VM Hardware via the Proxmox web GUI.

If you run pfSense (with is based on FreeBSD), I think you should select OS Type Other in the VM Options instead of Linux 5.x-2.6.
 
LEESTEKEN, thanks for your help. I applied the changes you outlined. I'm getting another error.

GUI Error
kvm: -device vfio-pci,host=0000:04:01.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on: vfio 0000:04:01.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy TASK ERROR: start failed: QEMU exited with code 1

DMESG Error
[ 161.060751] vfio-pci 0000:04:01.0: enabling device (0000 -> 0003) [ 161.098825] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:04:01.0)) vs. 00000080 (ehci_hcd:usb1)

VM .conf Contents
boot: order=ide0;ide2 cores: 2 hostpci0: 0000:04:01 ide0: local-lvm:vm-102-disk-0,size=32G ide2: local:iso/pfSense-CE-2.5.1-RELEASE-amd64.iso,media=cdrom memory: 2048 meta: creation-qemu=6.1.0,ctime=1650291736 numa: 0 ostype: other scsihw: virtio-scsi-pci smbios1: uuid=33dbd778-6c3d-43b7-bdcf-c7428e4b3298 sockets: 1 vmgenid: f24af903-5380-43e3-94e0-088a570bf15e

IOMMU GROUPS
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) IOMMU group 10 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0) IOMMU group 11 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] IOMMU group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4] IOMMU group 12 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] IOMMU group 12 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2] IOMMU group 13 05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242] IOMMU group 14 06:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01) IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) IOMMU group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) IOMMU group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1) IOMMU group 2 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] IOMMU group 3 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba] IOMMU group 4 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] IOMMU group 5 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] IOMMU group 6 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0] IOMMU group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) IOMMU group 8 00:1c.3 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev d0) IOMMU group 8 03:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04) IOMMU group 8 04:01.0 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) IOMMU group 8 04:01.1 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) IOMMU group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)

I tried both 04:01.0 and 04.01.1 solo. I then further removed "all functions" from the one PCI NIC I added. I then removed "ROM Bar". All four met with the same error message. This was the .conf file afterward reducing down to the smallest coverage area.

boot: order=ide0;ide2 cores: 2 hostpci0: 0000:04:01.0,rombar=0 ide0: local-lvm:vm-102-disk-0,size=32G ide2: local:iso/pfSense-CE-2.5.1-RELEASE-amd64.iso,media=cdrom memory: 2048 meta: creation-qemu=6.1.0,ctime=1650291736 numa: 0 ostype: other scsihw: virtio-scsi-pci smbios1: uuid=33dbd778-6c3d-43b7-bdcf-c7428e4b3298 sockets: 1 vmgenid: f24af903-5380-43e3-94e0-088a570bf15e

I appreciate your help and fast replies.
 
Maybe some network stuff is using the network devices Try adding vfio-pci.ids=8086:1010 to the kernel parameters in GRUB to make sure no driver touches the Intel network card (and update-grub and reboot). And double check before starting the VM with lspci -nnks 04:01 that Kernel driver in use: vfio-pci appears for both functions.
 
So I added vfio-pci.ids=8086:1010 to /etc/default/grub like so
(That's where I assumed it should go.)

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio-pci.ids=8086:1010"

Updated and rebooted.

This is what 'lspci -nnks 04:01 produces
04:01.0 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) Subsystem: Compaq Computer Corporation NC7170 Gigabit Server Adapter [0e11:00db] Kernel driver in use: vfio-pci Kernel modules: e1000 04:01.1 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) [8086:1010] (rev 01) Subsystem: Compaq Computer Corporation NC7170 Gigabit Server Adapter [0e11:00db] Kernel driver in use: vfio-pci Kernel modules: e1000

I start the VM and I'm getting the same error message as before.

[ 409.593276] vfio-pci 0000:04:01.0: enabling device (0000 -> 0003) [ 409.631063] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:04:01.0)) vs. 00000080 (ehci_hcd:usb1)

At this point in the troubleshooting process I start to wonder if I'm missing something glaringly obvious. Cheers.
 
Perhaps I should alert you to the fact that there is an ethernet port on the motherboard already. I apologize if that's something I should have mentioned earlier.
 
That sounds wise. That was my original assumption based on dumb logic, but would have preferred to stay in the slot it's in. Hmmm. I'll have to open her up and see what kind of freedom of movement I've got. Thanks for your help. I imagine the next time you hear from me, the problem will be fixed.
 
OK, got my hands dirty during lunch. I've got the NIC attached to the VM and the VM to start. Thanks for your help in the process! The fix only came by unplugging a USB device and switching the NIC and graphics card slots. I've at least moved forward...

For better or worse I think we can mark this as finished, though I've got a sneaky suspicion that this setup isn't done being janky.

I appreciate you!
 
  • Like
Reactions: hmohr and leesteken
I had the same issue with a Supermicro X8DTL-i mobo and a PCI RTL8169 NIC. IRQ conflicts and only one PCI slot available on the mobo. I have the most recent BIOS (the OG version didn't include the following option) and I went into the BIOS and disabled the upper half of the USB functionality. I still needed the USB keyboard. So, after about 6 hours of reboots and attempting various IOMMU and other trickery, the Proxmox server booted and brought up the VM without crashing. I almost fell off my chair! So, the solution is (almost) hardware related and nothing to do with Proxmox. the DMESG errors ( Flags mismatch irq ) were probably the biggest clue.
I'm a complete newbie to Proxmox and of the 8 hours I've spent on Proxmox so far, 3/4 of the time has been dedicated to getting this one #$*&#$ thing working. Dog, meet bone. Mmmm .. bone.
 
  • Like
Reactions: leesteken
From my personal experience PCIExpress card build on top of Asmedia ASM1083 PCIExpress bridge is not work good in virtual environment.
I had bad experience with Firewire card build with VIA chipset VT6307, which require ASM1083 to work as PCIExpress card: this card passed to guest OS in KVM passthrough mode lead to unpredictable server reboot. Finally, I have replaced card on VT6307 chipset with more modern with VT6315N chipset (which not require ASM1083 as PCIe bridge), and from that moment everything started working perfectly.
 
You must log in or register to reply here.
Top Bottom