GPU + NIC passthrough fail

Turok

New Member
Feb 21, 2024
1
0
1
I hope I can provide the info required to help. I'm a little lost with this whole passthrough thing. Thanks in advance. I'm here to help you help me.

Purpose - Passthrough nvidia 1080ti GPU AND in other VM running truenas with Mellanox MT27520 Family [ConnectX-3 Pro] 10gb NIC

The error:
kvm: -device vfio-pci,host=0000:08:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:08:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

dmesg error:
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:08:00.0)) vs. 00000080 (vfio-intx(0000:01:00.0))

lspci:
01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
08:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

01:00.0 0300: 10de:1b06 (rev a1) (prog-if 00 [VGA controller])
Subsystem: 1458:374c
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau

01:00.1 0403: 10de:10ef (rev a1)
Subsystem: 1458:374c
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

08:00.0 0200: 15b3:1007
Subsystem: 15b3:000c
Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

IOMMU groups:
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] [8086:a780] (rev 04)
IOMMU group 10 00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11)
IOMMU group 11 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac0] (rev 11)
IOMMU group 12 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:7ac4] (rev 11)
IOMMU group 13 00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 [8086:7ab8] (rev 11)
IOMMU group 14 00:1c.1 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 [8086:7ab9] (rev 11)
IOMMU group 15 00:1c.4 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #5 [8086:7abc] (rev 11)
IOMMU group 16 00:1d.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #9 [8086:7ab0] (rev 11)
IOMMU group 17 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a88] (rev 11)
IOMMU group 17 00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
IOMMU group 17 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
IOMMU group 17 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
IOMMU group 18 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
IOMMU group 18 01:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU group 19 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:a700] (rev 01)
IOMMU group 20 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU group 21 05:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E18 PCIe4 NVMe Controller [1987:5018] (rev 01)
IOMMU group 22 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-LM [8086:125b] (rev 06)
IOMMU group 23 07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-LM [8086:125b] (rev 06)
IOMMU group 24 08:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:a70d] (rev 01)
IOMMU group 3 00:06.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port [8086:a74d] (rev 01)
IOMMU group 4 00:0a.0 Signal processing controller [1180]: Intel Corporation Raptor Lake Crashlog and Telemetry [8086:a77d] (rev 01)
IOMMU group 5 00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller Intel Corporation [8086:a77f]
IOMMU group 6 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
IOMMU group 6 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
IOMMU group 7 00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)
IOMMU group 7 00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #1 [8086:7acd] (rev 11)
IOMMU group 7 00:15.2 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #2 [8086:7ace] (rev 11)
IOMMU group 8 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
IOMMU group 8 00:16.3 Serial controller [0700]: Intel Corporation Device [8086:7aeb] (rev 11)
IOMMU group 9 00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)

grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

/etc/modules :
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd]

/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/modprobe.d/pve-blacklist.conf
blacklist nvidiafb

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1b06,10de:10ef,15b3:1007 disable_vga=1


Ask every other output you need, i can't figure it out this issue by myself...
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!