NS + EW Router/Firewall implementation on Proxmox Cluster

[juliosene]

Hello!

I love the idea of SDNs on Proxmox Cluster. I believe that SDNs will evolve fast.

After trying some solutions to optimize my usage of Proxmox SDNs and improve the flexibility and security of my environment, I've decided to implement a solution based on the two routers-firewalls concept, also known as North-South + East-West implementation. With this architecture, I intend to use SDNs to segment my network and make implementation easy, but also have access to all features that a dedicated router/firewall solution has available.

The idea is simple: North-South Router/Firewall will take care of external (Internet/WAN) to internal network (local). East-West will handle internal traffic and network segmentation. Besides the fact that there are more appliances to manage, this architecture has some advantages:

• attacks will affect just one appliance, reducing effects and making diagnostics faster.

• changes on EW-router will not affect internal users connection with internet.

• ownership of NS-router and ES-router is not necessarily the same. Example: Security Team can be owner of NS-router and Infrastructure Team owns the ES-router.

• Proxmox SDNs will be managed and protected by recognized router and firewall solutions that will add value to the final architecture, especially for small and medium business implementations.

The final architecture is like this:





My option for master-backup solution intends to guarantee a service 100% disponible. For most use cases, especially when you are not using a Proxmox Cluster, a single EW-router will be good enough.

Some requirements of my architecture:
1 – As a starting point, everyone can talk to everyone. It means that a user on Internal Network has access to VMs on VNet Y or VNet X and vice-versa.
2 – No NAT. Internal traffic must use the machine's IP without any kind of translation.
3 – Block private networks and block bogon networks must be disable on EW-router Internal Network (WAN) interface.
4 – Changes to the NS–router must be minimized. I decided to route back a larger subnet (/17) and use smaller subnets (/24) to my VNets.
5 – Blocks must be implemented by firewall rules on EW-router. Example: for cloudflared public exposed network, I'd added a block Internal Network rule on this interface.


I'm a PFSense user, but I decided to test OpenSense to implement EW-router. The master-backup solution was implemented following this guide: Don't Use A Firewall, Use 2! OpnSense High Availability Guide (youtube.com)


If you need more details or even a step-by-step guide about how to implement this architecture, please, leave a reply here and I will do my best to help.

 

[esi_y]

So this is one of the (probably more elaborate) meaningful ways to do things properly.

I would just point out what puzzles me.

1 – As a starting point, everyone can talk to everyone. It means that a user on Internal Network has access to VMs on VNet Y or VNet X and vice-versa.

This is a nightmare. It is almost like existing firewall bug in PVE. Any firewall (including internal) with default ruleset that allows everything is a liability.

3 – Block private networks and block bogon networks must be disable on EW-router Internal Network (WAN) interface.

This is something very *sense centric, I suppose. You should really only have rules that allows what's necessary, the moment you start "disabling" some blanket block rules, you are assuming nothing you plug into the network can ever go rogue.

If you need more details or even a step-by-step guide about how to implement this architecture, please, leave a reply here and I will do my best to help.

If you assume everything connected from the inside is trusted, then of course the above makes sense, but that is not the real world scenario.

The other pet peeve - virtualised solutions are fine, but virtualising them together (on the same hardware) and (also) with the rest of the VMs is adding risks you better would not want to have, adding complexity and unlikely increasing availability. I would rather have non-HA setup, but dedicated hardware one (it will be more reliable). Finally, this seems very IPv4 centric, in a homelab setup, you end up with NAT over NAT (and possibly even CGNAT).

PS There are also more routing options out there than *Sense, this concept should not be limited to those.

If you are happy with your setup, you can however, completely ignore me (it's better than what you get from PVE already).

One thing I liked about your post though is that you did go through the extra thinking of what a proper firewall approach would be, instead of taking something ready-made out-of-the-box out of convenience.

 

[juliosene]

Hi esi_y! Thanks for your comment.

Nightmare -> Depends on what you are dreaming about.

My goal here is to propose an architecture to improve the usage of SNDs on Proxmox. For sure you can keep this architecture and implement a “Deny all” perspective. To be honest, I've decided to use “everyone can talk to everyone” as a starting point because it is a challenge when you use a firewall. As default, a *Sense (and may others) blocks everything from WAN -> LAN and LAN -> WAN is a NAT.

One important point to mention is that, even with “everyone can talk to everyone”, you will be adding an important point in terms of security perspective: observability. You can simply add log to a “pass” rule and understand the traffic before blocking something.

Anyway, the architecture can be used with any routing/firewall strategy.

Other flavors of firewall can use different names for it, but it is important to see if some default rule is blocking private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). If you are using your EW router WAN interface connected to your internal network that uses, for example, 10.0.0.1/24 and you must have access to a server locate into an SDN using 192.168.0.1/24 range, this communication is blocked by default by those rules.

Again, even with “everyone can talk to everyone”, this solution will add, at minimum, connectivity and observability.

The idea here is avoiding NAT over NAT. Thats the reason to disable the default rules blocking for private networks IP ranges. Only NW router (main router) will perform NAT. If you are using real IPs in your network, a different network segmentation will be necessary.

For sure there are many ways to improve it! You can change firewall rules restricted ones, you can add web proxy and WAF, and any other feature that *Sense has as plugins. Also, using external dedicated firewall hardware is an improvement. However, “Perfect is the enemy of good” and doing the best keeping what you have is an art.

The following diagram represents a simple version of the architecture that I'm using. Again: The rules are yours! Even if you will not use the full potential of this structure at the first moment, it is a good starting point and will reduce friction to evolve your environment in terms of network and security compared to a simple Proxmox installation.

As an example, in my case, I will change the rules for VNet X blocking any access from this network to my Internal Network or to VNet Y. I will use VNet X as a subnet to put my public exposed services and CloudFlareD as the CloudFlare tunnel endpoint. Internal users will have access to those services via the Internal Network, without any NAT. External users will have access via CloudFlare Tunnel. Even if some services were compromised by a hacker attack, to have access to my Internal Network will be much more difficult.

The idea here is to add a new perspective to Proxmox SDNs usage. If you can add a pair of F5 firewalls as EW protection, please, do it! But if you are running a single PVE installation on your home lab or a simple 3 machines cluster in an SMB company, be aware that adding a single or redundant EW *Sense router to your environment will add, at minimum, better observability and connectivity opening a lot of possibilities to improve your structure in terms of security with low costs (maybe zero costs).

This is an overview of the network architecture that I'm creating here.

 

[Stereoscope]

Absolutely intriguing. No idea how this works but I'll be studying this and will hopefully understand it enough to maybe use something similar for my setup. Thanks for sharing.
Just a question, what is VIP? Is that a Virtual IP? How does that work?

 

[juliosene]

Just a question, what is VIP? Is that a Virtual IP? How does that work?

Yes, VIP stands for Virtual IP. In routing systems like OPNSense, the VIP is important for maintaining a single routing IP address, even if two physical or virtual machines are performing as a router (usually designated as master and backup). This method ensures that the default gateway IP of the network remains unchanged, even if the machines alternate in being the master. It's important to remember that, in addition to the VIP, each machine needs to have its own individual IP.


It is important to remember that you don't need to start with a structure as complex as the one I created. My goal was to create a lab that replicates corporate use scenarios with high availability and control.

In most scenarios, router redundancy wouldn't be necessary. This alone significantly reduces the solution's complexity. By keeping the router VM in a backup routine and configuring Proxmox's HA to ensure it remains always active, it would already be a highly efficient and sufficient solution for most cases, even for non-productive corporate environments (development, quality control, homologation).

The idea of using a VM as a router to access virtual networks in Proxmox aims to provide greater control and freedom in the use of SDNs.

A common use case is to have a systems development team that needs access to one environment (DEV env) but cannot access others. By creating an SDN for the development team and segmenting access on Proxmox resources, this would be possible. However, routing for Proxmox SDNs becomes much more controlled if you have a VM with a system focused on this role, acting as a router/firewall. This is why I propose using a solution like OPNSense for this role (PFSense is also a great alternative). I created a simplified diagram to make the solution's objective clearer.




Remember that this type of solution only makes real sense in a Proxmox cluster, because on a single machine the native Proxmox Simple SDN Zone solution already works. However, in a cluster, the Simple SDN Zone creates isolated subnets on each host, not a unified subnet between them. In this case, the solution with dedicated router software in a VM comes in handy. In this case, we need to use VLAN-type SDN Zone and attach an interface to the subnet under this zone in OPNSense. With this, OPNSense can connect the VLAN network to the standard network, which also needs to have an interface associated with it on the firewall.

Remember that your main router needs to recognize the subnet networks. Therefore, you need to add a static route on this router, indicating that the IP ranges of the subnets (in this case, the subnets VNet X and VNet Y) have as their gateway the external static IP of the OPNSense (belonging to the default Proxmox local network).

 

[Stereoscope]

Thanks for the explanation, VIP now makes sense to me. Still trying to process everything else.