Passthrough to OPNSense VM Crashes Sytem

[fbeye]

Hello

After some Google research it seems this has been brought up a few times but their solutions are apparently not my solutions.
I have an HP Proliant 380 Gen9. In Bios, VT-D (?) is enabled and on Proxmox I did follow the howto to enable passthrough and I get 'DMAR: IOMMU enabled' so apparently that aspect works.

I create the VM and add no network to it. When I am done with it, I do not start it but go to hardware. On the HP there are 4 NIC's and assuming they are in order, I see only 4 broadcom NIC's available to add. I assume #1 is the one already in use as proxmox itself, so I add #2 and #3. When I start, whole thing crashes.
I did this withOUT adding any vbr0 or 1 etc, just left the NICs under main proxmox alone. I then went in and added a bridge to each NIC and re-added them, still crashes.
Not sure what I am missing

 

[fbeye]

By the way, if I do not "need" passthrough and can simply add 2 nics to it normally, I am open to that, it just seems OPNSense would work better with less configurations on the NIC that way as opposed to passthrough. But if I am wrong, I can do it however.

 

[fbeye]

Well, I did it as adding NIC instead of PCI, and added vmbr1 and vmbr2. Seems to be installing fine and as it should. I swear I did it by adding PCI before, but maybe not.

 

[Dunuin]

I would check the IOMMU groups. Crashing host when starting the VM usually is caused by stealing too much hardware from the host, as PCI passthrough will always passthrough the whole IOMMU group with all its members and not the single device/function you have selected.

 

[fbeye]

Not sure if this has any relevance;

lspci | grep Ethernet
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

find /sys/kernel/iommu_groups/ -type l | grep 02
/sys/kernel/iommu_groups/55/devices/0000:80:02.2
/sys/kernel/iommu_groups/53/devices/0000:80:02.0
/sys/kernel/iommu_groups/5/devices/0000:00:02.2
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/56/devices/0000:80:02.3
/sys/kernel/iommu_groups/54/devices/0000:80:02.1
/sys/kernel/iommu_groups/34/devices/0000:02:00.2
/sys/kernel/iommu_groups/34/devices/0000:02:00.0
/sys/kernel/iommu_groups/34/devices/0000:02:00.3
/sys/kernel/iommu_groups/34/devices/0000:02:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:02.3
/sys/kernel/iommu_groups/4/devices/0000:00:02.1

I mean, my thought was get the ethernet's listed, then see what group they are in, then post those among that group.

Not sure why they are not in order, but being that eno1 is the only one UP of the 4, I would have to say that en01 is 02:00.0. eno2 is 02:00:1 and so on.
In my VM, I was trying to add eno2 and eno3, which would be 02:00.1 and 02:00.2.
That is all the info I know or have, at this time.
Does my nd output suggest why my system freezes?

 

[Dunuin]

All 4 functions are in the same IOMMU group. So when passing though anyone of those ports, you will passthough all 4 of them.

 

[fbeye]

When you say function, you mean 4 in the 02:00:* ?
Is there a way to simply only allow the 2 I want?
I bet that is why it locks up. Prox does not lock up, as locally I still am able to connect via console, but everything network. So I assume it is passing through the working NIC as well, making it lock up.

 

[Dunuin]

The "*"-part of the "0000:02.00.*" is the function. Not all NICs will allow you to passthrough individual ports. If the NIC would allow that, than each function would have got it's dedicated IOMMU group.

 

[fbeye]

I do not understand how I did it before then! My MB fried and I replaced it, so I said WTH and redid the whole Proxmox.

 

[fbeye]

I just set it like I said earlier, create bridges for both NIC's and I did indeed get it working. Works fine so, I am happy.
Thank you for pointing me in directions that allowed me to look deeper.