Proxmox with OPNsense as Router/GW

[EugenMayer]

There are a lot of ressources online, old legacy, handling performance issues, partial multi-NIC setups and all this, but nothing is comprehensive, rather a lot of unfinished ones.

I try to not repeat the whole story i posted here https://stackoverflow.com/questions/44118442/proxmox-with-opnsense-as-firewall-gw-routing-issue with hopefully transparent drawings about the network layout.

Right now, i ask myself, if any proxmox experts actually can crack this?

a) First off, how to get proper "symetric" routing - it seems like routing is a bit off in this setup. Any particular error i made?

b) E.g. considering i have only this one NIC with one IP and 2 additional ips routed in my MAC. Since i have a bridged setup, those arrive at vmbr0 which my OPNsense is attached to. Should i now create Virtual IPs and multiple WANs in opnsense and attache my rancher VM as a 1:1 NAT in opnsense - will this actually work ( similar to , but there is no virtualization in place

)

c) Does shorewall harm here - would say no, i can see any blocks i do not want? Do i probably miss masq in Shorewall for the internal network?

I am really making up my mind since this topic seems to be quiet interesting for a lot of people and i would like to have at least one working solution.

HINT: In case people will ask "why dont i use the internal FW" - opnsense is a lot more in the case. It will provide a tinc-mesh network between several proxmox servers for the private LAN. It will provide IPsec support to connect into the private LAN using mobile clients. It will provide tinc site2site support to connect offices into the private lan. It offers DHCP and dynamic DHCP based DNS for all VMs so ease up bootstrapping a lot. There is a lot more to add here - nothing Proxmox can deal with or should deal with.

 

[manu]

had a look at your schema

for B) and your external IP3. I would add this IP to pfSense/Opnsense by itself, and then do port forwarding on the firewall.

C) Do i probably miss masq in Shorewall for the internal network?
do you mean dnsmasq here ? In any case if you use a DHCP/DNS server for you internal lan, it should run on OpnSense, more specially it should bind on the NIC 2 LAN of psense, so every machine in the vmbr30 have access to it, and no one except those

 

[EugenMayer]

b) yes, thought / focussing on this already, so virtual IP in opnsense + 1:1 nat, right o n that
c) not dnsmasq, but http://shorewall.org/manpages/shorewall-snat.html - so source natting

Currently working on PCI nic passthrough to ease things up / having a better performance.

If that fails i actually would move all IPs into OPNsense, even the one from proxmox https://stackoverflow.com/a/44129112/3625317 and operate from there. But slowly i am getting there

 

[manu]

C) why don't you do this on pfsense ? personnally I would rather have have a single firewall taking care of all the VM access
I would then assign the IP adress on the WAN port of psense, do port fortwarding to reach the VM sitting in vmbr30, and do source nat on pfsense again

 

[EugenMayer]

C) will be gone ( shorewall ) when PCIpassthrough is done - since it will obv make 0 sense as you mention