[SOLVED] Proxmox and OPNsense setup fails

[NevadaTech]

What I'm trying to do
* build a small Internet facing Proxmox box
* run OPNsense as virtual guest #1 for firewall/router services
* run virtual guest #2 Remotely behind OPNsense (10.12.99.10/24)
* run virtual guest #3 chat server behind OPNsense (10.12.99.20/24)
* run virtual guest #4 ftp server behind OPNsense (10.12.99.30/24)
* the servers/services can be anything, these are just examples, I understand each server has it's own firewall/routing concerns

My setup
* I have public static IP of 11.22.33.218/28 with a gateway of 11.22.33.209
* the server has an HP NC365T quad network card and a mobo built-in Realtek nic
* Proxmox is v8.1.4

My plan
* vmbr0 = NC365T port 0, enp6s0f0, infrastructure LAN, no Internet access needed (10.12.77.5/24)
* vmbr1 = NC365T port 1, enp6s0f1, unused
* vmbr2 = NC365T port 2, enp6s0f2, LAN, where all of the servers/services will live, vmbr2 assigned IP range (10.12.99.0/24) to port (enp6s0f2)
* vmbr3 = NC365T port 3, enp6s0f3, WAN, vmbr3 assigned to the port only (enp6s0f3), no IP address assigned via Proxmox
* OPNsense has (2) nics assigned
- vmbr2 used for LAN, inside OPNsense as vtnet1 as 10.12.99.1
- vmbr3 used for WAN, inside OPNsense as vtnet0 with 11.22.33.218/28

The problem is I can't get out to the Internet through OPNsense. Some things I've tried
* changing NIC port 3 (WAN) to PCI device, it locks up Proxmox
* using the built-in NIC to PCI device, it locks up Promox > am I missing something here?
* enabling and disabling the TCP hardware offloading
* allowing private IP segments to pass
* assigning public static IP to vmbr3, that should fail because I'm assigning an IP to the physical and again to the virtual OPNsense vtnet0
* I've seen some references to masquerading and iptables, I don't think they apply to my setup, correct?

Lastly, I'm not committed to this setup. If my basic plan is wrong please say so with a better design.

 

[NevadaTech]

Also, I know Proxmox has some firewall capabilities. I'm choosing OPNsense because I am more familiar with it. And I'd like to build a VPN to the LAN segment from outside.

 

[NevadaTech]

More variations for the answer...

* does anyone see anything wrong with the plan
* do you have OPNsense/pfSense running in a similar fashion, if so, what is your layout

 

[NevadaTech]

* allowing private IP segments to pass

So this setting is wrong. Some of the other post/setups that referred to this were router behind router (double NAT) setups. If your WAN DHCP gives you a 192.168.x.x or 172.(16-32?).x.x or 10.x.x.x address then you do need it. Most cases don't.

 

[NevadaTech]

While poking around last night I noticed this
* I still couldn't ping my ISP gateway
* when I looked at the streaming firewall logs there was a bunch of traffic hitting the router, the traffic was a mix of private and public IP; that was a big help, it kinda said the basic plan/setup was correct

I have several private IPs from Cox. My cablemodem feeds a switch that feeds a couple different routers. I think at some point the cablemodem tied the 11.22.33.218 IP to a specific interface on Proxmox. All of my moving and testing was for nil since the cablemodem said only this interface (MAC address) is allowed to this IP.

FIX: reboot the cablemodem; OPNsense picked up the 11.22.33.218 IP and I could ping the ISP gateway; so essentially the fix was an IT Crowd solution (try turning it off and back on again, you're welcome); sigh