NS + EW Router/Firewall implementation on Proxmox Cluster

[juliosene]

Hello!

I love the idea of SDNs on Proxmox Cluster. I believe that SDNs will evolve fast.

After trying some solutions to optimize my usage of Proxmox SDNs and improve the flexibility and security of my environment, I've decided to implement a solution based on the two routers-firewalls concept, also known as North-South + East-West implementation. With this architecture, I intend to use SDNs to segment my network and make implementation easy, but also have access to all features that a dedicated router/firewall solution has available.

The idea is simple: North-South Router/Firewall will take care of external (Internet/WAN) to internal network (local). East-West will handle internal traffic and network segmentation. Besides the fact that there are more appliances to manage, this architecture has some advantages:

• attacks will affect just one appliance, reducing effects and making diagnostics faster.

• changes on EW-router will not affect internal users connection with internet.

• ownership of NS-router and ES-router is not necessarily the same. Example: Security Team can be owner of NS-router and Infrastructure Team owns the ES-router.

• Proxmox SDNs will be managed and protected by recognized router and firewall solutions that will add value to the final architecture, especially for small and medium business implementations.

The final architecture is like this:





My option for master-backup solution intends to guarantee a service 100% disponible. For most use cases, especially when you are not using a Proxmox Cluster, a single EW-router will be good enough.

Some requirements of my architecture:
1 – As a starting point, everyone can talk to everyone. It means that a user on Internal Network has access to VMs on VNet Y or VNet X and vice-versa.
2 – No NAT. Internal traffic must use the machine's IP without any kind of translation.
3 – Block private networks and block bogon networks must be disable on EW-router Internal Network (WAN) interface.
4 – Changes to the NS–router must be minimized. I decided to route back a larger subnet (/17) and use smaller subnets (/24) to my VNets.
5 – Blocks must be implemented by firewall rules on EW-router. Example: for cloudflared public exposed network, I'd added a block Internal Network rule on this interface.


I'm a PFSense user, but I decided to test OpenSense to implement EW-router. The master-backup solution was implemented following this guide: Don't Use A Firewall, Use 2! OpnSense High Availability Guide (youtube.com)


If you need more details or even a step-by-step guide about how to implement this architecture, please, leave a reply here and I will do my best to help.

 

[esi_y]

So this is one of the (probably more elaborate) meaningful ways to do things properly.

I would just point out what puzzles me.

1 – As a starting point, everyone can talk to everyone. It means that a user on Internal Network has access to VMs on VNet Y or VNet X and vice-versa.

This is a nightmare. It is almost like existing firewall bug in PVE. Any firewall (including internal) with default ruleset that allows everything is a liability.

3 – Block private networks and block bogon networks must be disable on EW-router Internal Network (WAN) interface.

This is something very *sense centric, I suppose. You should really only have rules that allows what's necessary, the moment you start "disabling" some blanket block rules, you are assuming nothing you plug into the network can ever go rogue.

If you need more details or even a step-by-step guide about how to implement this architecture, please, leave a reply here and I will do my best to help.

If you assume everything connected from the inside is trusted, then of course the above makes sense, but that is not the real world scenario.

The other pet peeve - virtualised solutions are fine, but virtualising them together (on the same hardware) and (also) with the rest of the VMs is adding risks you better would not want to have, adding complexity and unlikely increasing availability. I would rather have non-HA setup, but dedicated hardware one (it will be more reliable). Finally, this seems very IPv4 centric, in a homelab setup, you end up with NAT over NAT (and possibly even CGNAT).

PS There are also more routing options out there than *Sense, this concept should not be limited to those.

If you are happy with your setup, you can however, completely ignore me (it's better than what you get from PVE already).

One thing I liked about your post though is that you did go through the extra thinking of what a proper firewall approach would be, instead of taking something ready-made out-of-the-box out of convenience.

 

[juliosene]

Hi esi_y! Thanks for your comment.

Nightmare -> Depends on what you are dreaming about.

My goal here is to propose an architecture to improve the usage of SNDs on Proxmox. For sure you can keep this architecture and implement a “Deny all” perspective. To be honest, I've decided to use “everyone can talk to everyone” as a starting point because it is a challenge when you use a firewall. As default, a *Sense (and may others) blocks everything from WAN -> LAN and LAN -> WAN is a NAT.

One important point to mention is that, even with “everyone can talk to everyone”, you will be adding an important point in terms of security perspective: observability. You can simply add log to a “pass” rule and understand the traffic before blocking something.

Anyway, the architecture can be used with any routing/firewall strategy.

Other flavors of firewall can use different names for it, but it is important to see if some default rule is blocking private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). If you are using your EW router WAN interface connected to your internal network that uses, for example, 10.0.0.1/24 and you must have access to a server locate into an SDN using 192.168.0.1/24 range, this communication is blocked by default by those rules.

Again, even with “everyone can talk to everyone”, this solution will add, at minimum, connectivity and observability.

The idea here is avoiding NAT over NAT. Thats the reason to disable the default rules blocking for private networks IP ranges. Only NW router (main router) will perform NAT. If you are using real IPs in your network, a different network segmentation will be necessary.

For sure there are many ways to improve it! You can change firewall rules restricted ones, you can add web proxy and WAF, and any other feature that *Sense has as plugins. Also, using external dedicated firewall hardware is an improvement. However, “Perfect is the enemy of good” and doing the best keeping what you have is an art.

The following diagram represents a simple version of the architecture that I'm using. Again: The rules are yours! Even if you will not use the full potential of this structure at the first moment, it is a good starting point and will reduce friction to evolve your environment in terms of network and security compared to a simple Proxmox installation.

As an example, in my case, I will change the rules for VNet X blocking any access from this network to my Internal Network or to VNet Y. I will use VNet X as a subnet to put my public exposed services and CloudFlareD as the CloudFlare tunnel endpoint. Internal users will have access to those services via the Internal Network, without any NAT. External users will have access via CloudFlare Tunnel. Even if some services were compromised by a hacker attack, to have access to my Internal Network will be much more difficult.

The idea here is to add a new perspective to Proxmox SDNs usage. If you can add a pair of F5 firewalls as EW protection, please, do it! But if you are running a single PVE installation on your home lab or a simple 3 machines cluster in an SMB company, be aware that adding a single or redundant EW *Sense router to your environment will add, at minimum, better observability and connectivity opening a lot of possibilities to improve your structure in terms of security with low costs (maybe zero costs).

This is an overview of the network architecture that I'm creating here.