PCIe Passthrough No irq handler for vector

[UnixChump]

Hello. I'm having problems with my NIC PCIe passthrough setup and I'm looking for support.

The VM network interface flips on and off after being left alone for a few hours or days. Network load might trigger the problem, considering I tested the hypervisor with a nearly identical VM that did not get much load and it did not have this problem. This "no irq handler for vector" warning gets spammed every few seconds to dmesg until the VM gets rebooted.

dmesg from Proxmox:

[Fri Mar 22 10:18:37 2024] __common_interrupt: 33.34 No irq handler for vector

--My Setup--

• Motherboard: Supermicro X10 SRD-F
• Device being passthrough: Intel X550
• Proxmox Version: 7.2-3
• Proxmox Kernel: 5.15.39-4
• Guest OS: Centos 7.6.1810
• Guest Kernel: 5.8.0-1.el7
• Proxmox has ZFS for the boot drives, which requires it to run in EFI mode. I'm using systemd to boot.

root@proxmox-23:~# qm config 102
balloon: 0
boot: order=scsi1
cores: 36
cpu: host
hostpci0: 0000:02:00.1
memory: 40960
meta: creation-qemu=6.2.0,ctime=1711074617
name: LoadedVM
numa: 0
ostype: l26
scsi1: local-zfs:vm-102-disk-1,size=270G
scsihw: virtio-scsi-pci
smbios1: uuid=9199f887-61c8-4e1a-98cb-132e66d9b62b
sockets: 1
vmgenid: 28640bb1-8647-4bde-92f4-5785cf82c3b7

02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)      <----- Management device.
        Subsystem: Super Micro Computer Inc Ethernet Controller 10G X550T [15d9:0903]
02:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)      <----- Passthrough device.
        Subsystem: Super Micro Computer Inc Ethernet Controller 10G X550T [15d9:0903]

root@proxmox-23:~# uname -a
Linux proxmox-23 5.15.39-4-pve #1 SMP PVE 5.15.39-4 (Mon, 08 Aug 2022 15:11:15 +0200) x86_64 GNU/Linux

root@proxmox-23:~# ethtool -i enp2s0f1
driver: ixgbe
version: 5.15.39-4-pve
firmware-version: 0x800007f6, 1.1681.0
expansion-rom-version:
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

/etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt intremap=no_x2apic_optout

/boot/efi/loader/entrie/boot/efi/loader/entries/proxmox-5.15.39-4-pve.conf
title    Proxmox Virtual Environment
version  5.15.39-4-pve
options  root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt intremap=no_x2apic_optout
linux    /EFI/proxmox/5.15.39-4-pve/vmlinuz-5.15.39-4-pve
initrd   /EFI/proxmox/5.15.39-4-pve/initrd.img-5.15.39-4-pve

/etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:1563

class    │ device │ id           │ iommugroup │ vendor │ device_name                                                                                          │ mdev │ subsystem_device │ subsystem_device_name │ subsystem_vendor │ subsystem_vendor_name    │ vendor_name             │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼──────────────────────────┼─────────────────────────┤
│ 0x020000 │ 0x1563 │ 0000:02:00.0 │         38 │ 0x8086 │ Ethernet Controller 10G X550T                                                                        │      │ 0x0903           │                       │ 0x15d9           │ Super Micro Computer Inc │ Intel Corporation       │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼──────────────────────────┼─────────────────────────┤
│ 0x020000 │ 0x1563 │ 0000:02:00.1 │         39 │ 0x8086 │ Ethernet Controller 10G X550T                                                                        │      │ 0x0903           │                       │ 0x15d9           │ Super Micro Computer Inc │ Intel Corporation       │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼──────────────────────────┼─────────────────────────┤

pveversion:

proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[BobhWasatch]

You generally can't pass-through only one function of a multi-function device. Are you sure the two ports are in different IOMMU groups? I can't tell from what you provided because you didn't show both ports.

If there's some reason this VM actually needs it's own network device rather than just being attached to the regular bridge, you could create a new bridge just for it and use the second port as the upstream. And if you say you can't do that because of "performance" I'll have to throw you into the "ricer" bin.

 

[leesteken]

proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)

Maybe upgrade to the latest 7.4 (so you can eventually upgrade to the latest 8.x) to check if this has already been fixed? You'll get security updates for free and at least you'll be on a supported version.

 

[UnixChump]

You generally can't pass-through only one function of a multi-function device. Are you sure the two ports are in different IOMMU groups? I can't tell from what you provided because you didn't show both ports.

If there's some reason this VM actually needs it's own network device rather than just being attached to the regular bridge, you could create a new bridge just for it and use the second port as the upstream. And if you say you can't do that because of "performance" I'll have to throw you into the "ricer" bin.

I've updated my post with both network devices (edit is pending mod approval at the time of writing this comment).
The management interface (0000:02:00.0) is in group 38.
The VM interface (0000:02:00.1) is in group 39.

I am attempting to tune software which only functions correctly with multiple IRQ lanes. I tried a setup with SR-IOV and standard virtual bridges, but it wasn't performing well.. In the past, I used PCIe passthrough on my old Citrix hypervisors for my application, and it worked perfectly. So now I'm trying to replicate my old Citrix setup performance-wise as much as possible on Proxmox.

Throw me in the ricer bin, whatever. It's the best solution I could find, and it worked (at least until this interrupt issue showed up).