Dual Proxmox + Dual Firewall Setup

[imothep77]

Hi All,

I would like to implement the following.

I've been fighting the whole weekend to set this up, with no success so far.

Initially, I wanted to use Sophos XG as my Firewall, but I reverted back to Pfsense for now as it seems easier to configure for a noob like me.

On my Netgear switch, let's keep aside for now the Internet connection as I set VLAN 78 on ports 1 and 2, port 1 untagged, as this is supposed to be connected to my ISP provider who can't self-assign a VLAN, and port 2 tagged, on the Prox-01 Protectli-like device, I set vmbr0 to use the same VLAN 78 and set the interface as VLAN aware... Let's also keep aside Prox-02, which is currently not connected to this network as this is my production machine atm.

The Protectli device has enough ports so I can use one for the WAN and another for the LAN, both connected to my switch. I set vmbr3 (port4) as VLAN aware.

On my PfSense VM, I set

• vmbr0.78 as my WAN interface
• vmbr3 as my LAN interface, DHCP setup with a 10.xx.xx.xx subnet

I verified the FW rules, and I have set "LAN net to all" as Pass.

I haven't set VLAN 70 yet as my management VLAN
When plugged to port 7 on my Netgear with my laptop with static IP, I'm able to access the switch web interface which address I set on the same subnet and the PfSense web page using the LAN interface. I am NOT able to reach my Prox-01 interface (same LAN subnet).

What am I doing wrong?

Thanks in advance for your precious advice.

Best

 

[imothep77]

I managed to resolve my few issues.

For the people who might want to configure a similar network, here are some of my lessons learned:

• The key thing here is to segregate WAN from LAN/MGMT on your managed switch. This was clearly understood, but not properly setup, see items below...
• LAN & MGMT VLANs should be set to different networks. My first mistake was to assign 2 different PfSense interface addresses to those VLANs but on the same subnet --> definitely not good.
• Implement VLAN on the interface you pass on to PfSense in Proxmox OR in PfSense VLAN configuration, NOT BOTH!! I was silly enough to pass VMBR1.78 as my WAN interface in PfSense (VTNET1) and set on top VLAN 78 in PfSense to serve as my WAN... likewise for the LAN interface.
• Check your cables - I really thought I was becoming crazy, I read so many articles, book passages and watches so many videos on VLANs and PfSense and Proxmox and I was sure I had it all set properly... but it still not working... until I discover I didn't have all the lights blinking in my switch... one cable was not properly plugged (the RJ45 plug lost his little plastic thing that normally holds it to its port, have actually several dirty cables like this one, if someone has a good shop site to get some new ones from, please let me know). I plugged the cable correctly, waited a few seconds... and voilà !!!
• To make sure FW is not coming into play yet for troubleshooting, make sure you set your rules - ON EACH INTERFACE - to pass all traffic from related VLAN to
• At this point in time, I had access to all my web ui's (proxmox, pfsense)
• WAIT - either if you're moving cables from port to port or after a configuration change on Proxmox or PfSense, give it time be taken into account
• Check that nothing in the BIOS is preventing Proxmox or PFsense from running smoothly - see issue resolution below - keep in mind that network is highly dependent on CPU in a virtualized environment.

I now have an issue with low speed between pfsense & proxmox.
When running iperf3 from proxmox shell to internet, I get the proper full WAN spped. When running from PFsense to proxmox host, I'm getting very poor speeds....

--> After 24h of troubleshooting, this was due to a wrong setup in my bios allowing passive t-states to throttle the CPU, i.e. to lower it's frequency depending on temp. I'm switching off my whole test lab during the night. I was not understanding, why, in the morning, everything was looking great, but not so after a couple of hours of playing around with it.... adding a new bullet point to the 'watch list'.


I'm continuing my thiing, next on :

• resolve speed isue
• setup proper fw rules (that might affect the above)
• create a proxmox HA cluster
• create a PfSense HA cluster
• Install PiHole (maybe dual HA cluster)
• etc..