How to passthrough Mellanox ConnectX-2 10GbE NIC to VM?

testcb00

Member
Jun 4, 2020
20
0
6
27
Hi everyone, recently I bought an old 1U X9 Supermicro server. I installed Proxmox VE 7.0 but I find that I am not able to passthrough the Mellanox ConnectX-2 10GbE NIC to my TrueNAS VM.

I follow to this guide https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough
Added
intel_iommu=on
to /etc/default/grub.

Added
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
to /etc/modules

Use
update-grub && update-initramfs -u -k all
to update the boot parameters

And
Code:
root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.021484] ACPI: DMAR 0x000000007E1D89D8 0000C4 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.021521] ACPI: Reserving DMAR table memory at [mem 0x7e1d89d8-0x7e1d8a9b]
[    0.102831] DMAR: IOMMU enabled
[    0.273054] DMAR: Host address width 46
[    0.273055] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.273063] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.273067] DMAR: RMRR base: 0x0000007f22c000 end: 0x0000007f23afff
[    0.273070] DMAR: ATSR flags: 0x0
[    0.273072] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.273075] DMAR-IR: IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.273077] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
[    0.273079] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.273081] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.273439] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.172207] DMAR: dmar0: Using Queued invalidation
[    2.186616] DMAR: Intel(R) Virtualization Technology for Directed I/O
[  880.222925] DMAR: DRHD: handling fault status reg 2
[  880.222964] DMAR: [DMA Write] Request device [05:00.0] PASID ffffffff fault addr 7fd3000 [fault reason 05] PTE Write access is not set

However, when I try to passthrough the NIC, the VM console show this error.
Code:
mlx4_core0: Unable to determine PCI device chain minimum BW

mlx4_core0: command 0x31 timed out (go bit not cleared)
mlx4_core0: NOP command failed to generate MSI-X interrupt IRQ 264)
mlx4_core0: Trying again without MSI-X
 
I modify the monitor to SPICE and adjust the console setting.
Now I can see more details on the console
It said "BIOS or ACPI interrupt routing problem", do it related to my Host (X9SRG-F) BIOS or it is the VM bios issue?

1632052135496.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!