[SOLVED] Proxmox freezes when starting any VM I add PCI Pass-through Devices to

[fbeye]

Hello
So I am wanting to make a VM and pass through 2 NIC's to it, for an OPNSense VM.
I am running an HPE Proliant DL380 w/ 4 OnBoard NIC's and I have an added 10Gbps NIC. I want NIC #4 and Add-On to pass through; when I add them and start the VM, no errors, I just lose all connectivity and it [Prox] freezes.

grep -e DMAR -e IOMMU
[    0.021167] ACPI: DMAR 0x000000007B7E7000 0002CE (v01 HP     ProLiant 00000001 HP   00000001)
[    0.021255] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e72cd]
[    0.553438] DMAR: IOMMU enabled
[    1.379700] DMAR: Host address width 46
[    1.379703] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.379716] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    1.379722] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.379730] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    1.379734] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[    1.379744] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[    1.379750] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[    1.379753] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[    1.379756] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[    1.379758] DMAR: ATSR flags: 0x0
[    1.379763] DMAR: ATSR flags: 0x0
[    1.379768] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.379772] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.379776] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.379778] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.379782] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.381041] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.681723] DMAR: No SATC found
[    1.681726] DMAR: dmar0: Using Queued invalidation
[    1.681736] DMAR: dmar1: Using Queued invalidation
[    1.727549] DMAR: Intel(R) Virtualization Technology for Directed I/O

dmesg | grep 'remapping'
[    1.379782] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.381041] DMAR-IR: Enabled IRQ remapping in x2apic mode

dmesg | grep -i vfio
[    5.314800] VFIO - User Level meta-driver version: 0.3

These are the 4 NIC,s I wanna pass-through the 4th one;

02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

And this is the 10Gbps;

81:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)

As I said, I just want those 2 NIC's pass through and I add them but the VM and Proxmox as a whole freezes. As far as configurations, that was it..

 

[leesteken]

As I said, I just want those 2 NIC's pass through and I add them but the VM and Proxmox as a whole freezes. As far as configurations, that was it..

Did you check your IOMMU groups: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation ? Devices in the same group cannot be shared between different VMs and/or the Proxmox host.

 

[gfngfn256]

As leesteken has pointed out you are probably passing through devices that are not IOMMU group-segregated.
You will probably find that the 4 O/B NICS are all in the same group. You could probably use the added 10gb NIC for the host access & passthrough the O/B (all four unfortunately) to the VM. This may or may not suit your needs.

 

[fbeye]

Ahhh that makes perfect sense! No that will not do so I will simply put in another NIC.. Reason is I need 1 NIC for the WAN port and a 10GBps for the LAN as Opnsense is the Firewall/Router.. Ok cool that makes a lot of sense. One question, I did see that website and reference but could not figure out what I was really looking at....

──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────────
│ 0x020000 │ 0x1657 │ 0000:02:00.0 │ 55 │ 0x14e4 │ NetXtreme BCM5719 Gigabit Ethernet PCIe
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────────
│ 0x020000 │ 0x1657 │ 0000:02:00.1 │ 55 │ 0x14e4 │ NetXtreme BCM5719 Gigabit Ethernet PCIe
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────────
│ 0x020000 │ 0x1657 │ 0000:02:00.2 │ 55 │ 0x14e4 │ NetXtreme BCM5719 Gigabit Ethernet PCIe
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────────
│ 0x020000 │ 0x1657 │ 0000:02:00.3 │ 55 │ 0x14e4 │ NetXtreme BCM5719 Gigabit Ethernet PCIe

I assume the Device 0x1657 and the 0000:02:00X is the same NODE meaning?

 

[gfngfn256]

You could use the following for a full IOMMU listing with ports etc.

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

 

[gfngfn256]

0000:02:00X is the same

No that is the port. 55 appears to be the IOMMU group for all those NICs.

 

[fbeye]

Amazing. I learned more in this 10 minutes than a long time. And this is also why it crashed cause when I started the VM, it essentially disconnected my NIC I was using for Prox itself.

 

[fbeye]

One last question... Now that I think of it, I recall my only spare NIC is indeed a 4 Port Card... When passing it through, do I need to pass through all 4 of those ports or being I am only using 1 of the 4 can I just pass that 1 through, and of course forfeit the other 3 from being used elsewhere. I know I will end up just passing it through but I guess my question is more about logistics.

 

[gfngfn256]

can I just pass that 1 through, and of course forfeit the other 3 from being used elsewhere

AFAIK yes. You will just lose those remaining NICs elsewhere.

 

[gfngfn256]

Maybe mark this thread as Solved. At the top of the thread, choose the Edit thread button, then from the (no prefix) dropdown choose Solved.

 

[fbeye]

Roger that, ty

 

[guruevi]

If you are passing through a device, you pass through the device. The number of ports on the device doesn’t matter.

Specifically with NIC you can connect virtual NIC to your individual hardware ports in software, you don’t have to pass them through as hardware.

Many modern NIC (I don’t know about the BCM chip you got) at least support SR-IOV, in that case you have to enable IOMMU/SR-IOV in your UEFI and then with the vendor tool you can create VF in hardware, provided your CPU works with it you can pass through a VF for a single port (or even a VLAN on the NIC), the difference being that the vNIC (virtual function) is done in hardware.

 

[fbeye]

If you are passing through a device, you pass through the device. The number of ports on the device doesn’t matter.

Specifically with NIC you can connect virtual NIC to your individual hardware ports in software, you don’t have to pass them through as hardware.

Many modern NIC (I don’t know about the BCM chip you got) at least support SR-IOV, in that case you have to enable IOMMU/SR-IOV in your UEFI and then with the vendor tool you can create VF in hardware, provided your CPU works with it you can pass through a VF for a single port (or even a VLAN on the NIC), the difference being that the vNIC (virtual function) is done in hardware.

Bare in mind I am very slow at this;

From what I believe you are saying, when I add a PCI Device and it shows for example the 4 Onboard NIC's, if I select [1] of the 4 it will still pass all 4 as it passes through the "device"?

I will look into the virtual NIC, I guess I was focused on passing through because they were to become the WAN and LAN specific devices within the VM for the virtual router and did not want any possibility of messing that up by still leaving it available on the Proxmox for other usage.

The final part, I am too much a noob to really understand that.

 

[leesteken]

If you are passing through a device, you pass through the device. The number of ports on the device doesn’t matter.

From what I believe you are saying, when I add a PCI Device and it shows for example the 4 Onboard NIC's, if I select [1] of the 4 it will still pass all 4 as it passes through the "device"?

Some network controllers have their ports in separate functions that actually are in separate IOMMU groups. Some network controllers show up as multiple devices (with one or multiple ports or functions). It depends on the PCIe layout on the add-in card and whether it has a PCIe multiplexer or switch with the right IOMMU capabilities.