How to passthrough Mellanox ConnectX-2 10GbE NIC to VM?

testcb00

Member
Jun 4, 2020
20
0
21
29
Hi everyone, recently I bought an old 1U X9 Supermicro server. I installed Proxmox VE 7.0 but I find that I am not able to passthrough the Mellanox ConnectX-2 10GbE NIC to my TrueNAS VM.

I follow to this guide https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough
Added
intel_iommu=on
to /etc/default/grub.

Added
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
to /etc/modules

Use
update-grub && update-initramfs -u -k all
to update the boot parameters

And
Code:
root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.021484] ACPI: DMAR 0x000000007E1D89D8 0000C4 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.021521] ACPI: Reserving DMAR table memory at [mem 0x7e1d89d8-0x7e1d8a9b]
[    0.102831] DMAR: IOMMU enabled
[    0.273054] DMAR: Host address width 46
[    0.273055] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.273063] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.273067] DMAR: RMRR base: 0x0000007f22c000 end: 0x0000007f23afff
[    0.273070] DMAR: ATSR flags: 0x0
[    0.273072] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.273075] DMAR-IR: IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.273077] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
[    0.273079] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.273081] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.273439] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.172207] DMAR: dmar0: Using Queued invalidation
[    2.186616] DMAR: Intel(R) Virtualization Technology for Directed I/O
[  880.222925] DMAR: DRHD: handling fault status reg 2
[  880.222964] DMAR: [DMA Write] Request device [05:00.0] PASID ffffffff fault addr 7fd3000 [fault reason 05] PTE Write access is not set

However, when I try to passthrough the NIC, the VM console show this error.
Code:
mlx4_core0: Unable to determine PCI device chain minimum BW

mlx4_core0: command 0x31 timed out (go bit not cleared)
mlx4_core0: NOP command failed to generate MSI-X interrupt IRQ 264)
mlx4_core0: Trying again without MSI-X
 
I modify the monitor to SPICE and adjust the console setting.
Now I can see more details on the console
It said "BIOS or ACPI interrupt routing problem", do it related to my Host (X9SRG-F) BIOS or it is the VM bios issue?

1632052135496.png