Proxmox/Opnsense/PCI Device w/PCI-Express - reboots take down PM node

[PenguinOrDevil]

Howdy and good morning/afternoon/evening/night,

Long time a listener, first-time caller. I've been using ProxMox in a dev/play environment in my home lab for about two years, and I love it. I've encountered an issue I've seen noted a few times but not answered. Please forgive me if my searching skills are lacking; I thought I was pretty comprehensive.

Long story short/TL;DR, Everything works fine; I get good throughput using `opnsense` virtualized as I'm using PCI Passthrough. It will run for weeks without any issues. The problem comes in when I reboot my `opnsense` VM. All the network ports, PCI Passthrough or otherwise, on my ProxMox node become 100% unresponsive until it comes back up, even the ports not being used by the `opnsense` VM. Additionally, and most concerning, the network becomes unresponsive forever if I shut down the VM. This is a headless machine, so I have taken the drastic step of power cycling to fix it. Here is my setup:

1. ProxMox, single host, as I mentioned, in a dev/play environment
2. Qotom brand 6 ethernet port fanless micro PC with i5-7200U, 32GB of memory, and some storage. One of these:
3. I have 3 VMs and two LXC containers on here, mostly my firewall/routing/pihole/Sensu monitoring setup and a Minecraft server for the kiddos.
4. Opnsense is using physical ports 1-3
   1. WAN
   2. LAN
   3. IOT (Because, in the immortal words of `The Offspring,` "You gotta keep'em separated." And I prefer physical segmentation over VLANs)
5. Other ports in use:
   1. 4 is in PCI Passthrough mode with identical configuration as the `opnsense` ports, but this VM is Linux. If I reboot this VM, there is no issue.
   2. 6 is a Linux Bridge with most of the VMs and how I get to the web interface for ProxMox.
6. Physical ports 2, 4, and 6 are all plugged into my switch.
7. Here is the config for the opnsense VM:
8. Each of the PCI Passthrough devices, for physical ports 1-3, are represented here as hostpci0-hostpci3, and all have a config similar to this:
9. Physical port 6, where most of my other VMs are getting to the network, is a Linux Bridge. This becomes important when I describe my testing. Here's its config:

In my testing, I'll reboot my `opnsense` VM after starting a constant ping to the PM node IP, every VM on the node, including the dedicated port for the Linux server with its own PCI Passthrough NIC, the `opnsense` VM, and outside the network to 8.8.8.8. At this point, I lose the ability to communicate to any physical network port during the duration of the reboot of the `opnsense` VM until it's back up.

I am at a loss; I've tried to go through documentation and this forum and haven't found a solution. I suspect I may have missed something basic or have a misunderstanding of one of the configuration items I've set, and I'd very much appreciate it if someone would help, or point out the issue I've created for myself.

Much appreciated!

Justin

 

[PenguinOrDevil]

Hey Y'all,

I never received a response on this and never found a solution. I ended up installing OPNsense on this hardware as a bare metal install. Of course, it works fine. I guess there is a bug in here, but I'm uncertain how best to add that as a bug now that I've redeployed it as bare metal.

The other workloads I had intended for this host have been moved to other hosts, as I needed to complete this testing involving OPNsense and move on. If I get motivated, I may try to recreate this issue and put in a bug.

If anyone has any questions, feedback, or solutions for the future, I will welcome them.