ProxMox, Vlan's and ACL Rules

[ianmud]

Background: I am a relative networking novice and what I have learned has been mostly by accident and preceded the advent of useful AI's so is quite cobbled together. I have somehow managed to set up what I believe to be a relatively secure network and am trying to be prudent about how I add ProxMox to the mix. I have it installed on the management network now, but am not willing to add more VM's until I can find a more secure way of proceeding.

Environment:
TP-Link Omada SDN software controlled network (running in ProxMox VM on Ubuntu 20.04)
Management Network (VLAN56) 192.168.56.xx (router, smart switch, access points, my pc)
Non_Critical Network (VLAN156) 192.168.156.xx
ProxMox Version 8.1.4 (at present, running in Management Network)
ACL rules:
1. A pair of rules that allow access to the entire network from vlan56 but that prevent access to that vlan56 from vlan156.
2. A set of rules port based rules that allow such things as printers on vlan156 to communicate with my personal computer that exists on vlan56

Here is what I think I would like to do (it may be that there is a better way, and if so, I am all ears):

1. Add a new VLAN, vlan256 and run just ProxMox on that VLAN (I don't need help with this)

2. Run my Omada Software controller in a Proxmox VM on vlan56 (management vlan). I am hoping for:
a. A pointer to a good primer for setting up the linux bridge and any ancillary requirements (it can be a gui based primer or a command line primer)
b. An explanation of what ACL Rules I should put in place that punches a hole into the management network (vlan56) from vlan256 (ProxMox vlan) but that doesn't expose more of vlan56 than is required to install the omada sdn software (on ubuntu 20.04) and communicate with the VM. So, for example, if I can limit access to specific protocols (UDP, TCP, etc...) and ports, that would be great.

3. Run all other VM's/LXC Containers on the vlan156 network.
a. I assume that item 2a will apply here as well
b. I also assume that 2b will apply here as well, but my main concern in this case isn't protecting the devices running on vlan156, but rather, protecting the proxmox server itself.

Any and all guidance will be appreciated.

 

[louie1961]

Personal preference but I prefer to not run my network/firewall/router software in a VM. Yes its possible, but every time you need to reboot your server for whatever reason, you will take down the internet for the rest of your home, assuming it all feeds off the same network or WAN. That being said, I don't really use ACLs as I think it is not as useful as a good VLAN setup. For background, I use pfSense as my firewall/router, and I have a managed switch between pfSense, my Proxmox nodes, my WAP, and my NAS machines. I have the following VLANs set up: trusted, guest, internet facing servers, IOT, Televisions, and management. Instead of ACLs I push different devices into appropriate VLANs. My WAP can have up to 8 SSIDs and each SSID can be tied to a specific VLAN. My PC, my wife's PC, one NAS and our printer is on the trusted VLAN. Kids and guests are all on Guest. All my publicly facing websites, Nexcloud instance and a NAS supporting all of those are on the "internet facing" VLAN. IOT is all the Ring devices, Ring alarm and the Honeywell thermostat. Televisions is all the devices with Roku sticks. Management is only for my WAP, switch, router/firewall and Proxmox nodes. Each VLAN has very restrictive firewall rules in pfSense, such that only the trusted VLAN can traverse to other VLANs. Each SSID is tied to one VLAN and have strong, unique passwords.

All of my proxmox nodes are on VLAN aware bridges and connected by trunk ports in the switch. I can set which VLAN I want each container or VM to be on inside of Proxmox. I do something similar within Docker and I actually run two different VMs for docker: one on the internet facing VLAN and one on the trusted VLAN. I also have two NAS boxes, one on the internet facing VLAN and one on the Trusted VLAN.

My internet facing websites all reach the internet via Cloudflare tunnels. Those all run in docker on the locked down internet facing VLAN

 

[ianmud]

Personal preference but I prefer to not run my network/firewall/router software in a VM. Yes its possible, but every time you need to reboot your server for whatever reason, you will take down the internet for the rest of your home, assuming it all feeds off the same network or WAN. That being said, I don't really use ACLs as I think it is not as useful as a good VLAN setup. For background, I use pfSense as my firewall/router, and I have a managed switch between pfSense, my Proxmox nodes, my WAP, and my NAS machines. I have the following VLANs set up: trusted, guest, internet facing servers, IOT, Televisions, and management. Instead of ACLs I push different devices into appropriate VLANs. My WAP can have up to 8 SSIDs and each SSID can be tied to a specific VLAN. My PC, my wife's PC, one NAS and our printer is on the trusted VLAN. Kids and guests are all on Guest. All my publicly facing websites, Nexcloud instance and a NAS supporting all of those are on the "internet facing" VLAN. IOT is all the Ring devices, Ring alarm and the Honeywell thermostat. Televisions is all the devices with Roku sticks. Management is only for my WAP, switch, router/firewall and Proxmox nodes. Each VLAN has very restrictive firewall rules in pfSense, such that only the trusted VLAN can traverse to other VLANs. Each SSID is tied to one VLAN and have strong, unique passwords.

All of my proxmox nodes are on VLAN aware bridges and connected by trunk ports in the switch. I can set which VLAN I want each container or VM to be on inside of Proxmox. I do something similar within Docker and I actually run two different VMs for docker: one on the internet facing VLAN and one on the trusted VLAN. I also have two NAS boxes, one on the internet facing VLAN and one on the Trusted VLAN.

My internet facing websites all reach the internet via Cloudflare tunnels. Those all run in docker on the locked down internet facing VLAN

Wow, that is quite the network. I am curious about whether you allow interaction between the many different vlans you have? Many devices/applications provide their own web interface for management but many don't. I am curious about whether/how you manage those that don't since you are not using ACL's.

Also, an advantage of using the Omada SDN is that if it fails or goes down for whatever reason, it has no impact on the network. Its purpose is to manage the configuration of these devices and track activity across the network. Once the devices are configured, it simply sits in the background and observes, so if it goes down, the network stays up.

 

[louie1961]

Wow, that is quite the network. I am curious about whether you allow interaction between the many different vlans you have? Many devices/applications provide their own web interface for management but many don't. I am curious about whether/how you manage those that don't since you are not using ACL's.

My trusted VLAN can go anywhere on the network. So I can see any web interfaces from my trusted VLAN. Also, all VLANs can go out to the internet. So for things like the web interface for my Ring cameras, I can always access those from anywhere on the web, even when I am away from home. All the other VLANs are locked down and cannot cross over/communicate with any other VLANs. Plus my trusted VLAN is the only one that can access the web interface for pfSense. I used to have to punch holes in the firewall rules to allow other VLANs to access my NAS, specifically my internet facing VLAN would have to be able to access my only NAS at the time which sits on my trusted VLAN. I decided for the sake of segregation, I would buy a second NAS and dedicate it to that external facing VLAN. Combine that with the Cloudflare tunnels and I no longer have any firewall rules to allow that cross VLAN communication from the internet facing VLAN. I also have no ports open on my firewall. It may not be the best way to do it, but it was the most expedient way I could come up with and not have to punch any holes in my firewall.

 

[louie1961]

Also, the nice thing about pfSense is I have Tailscale running on my pfSense box, and I advertise routes out through pfSense. So when I connect to tailscale with my laptop, regardless of where I am in the world, Tailscale allows me to operate as if I was at home. Some people don't trust Tailscale or Cloudflare tunnels, but I am not in that camp.

 

[myscha]

@louie1961
Would you mind sharing more details about you Proxmox network and your Docker network settings? Is it necessary to run multiple Docker instances or can the VLAN be selected for each container/stack?

 

[louie1961]

I run docker in two separate VMs, one in the 'trusted" VLAN, and one in the "untrusted" VLAN. So the VM that hosts things like my Heimdall dashboard, Uptime Kuma, Librespeed, Mealie, and most of my other docker containers runs in the trusted VLAN. I use MACVLAN networks so that my pfsense can assign IP addresses to these docker containers in my trusted VLAN as well. In the untrusted VLAN, I run all of the cloudflare tunnel connectors, and a couple of wordpress containers that I am experimenting with. They all get IP addressed from a different MACVLAN network and IP addresses from the pfsense box, but on the untrusted VLAN

This may not be the best way to do this, and I am still coming up the learning curve with docker, especially when it comes to mounting docker volumes on NFS shares and using a proxy like nginx. But from a security standpoint, anything facing the internet is pretty well isolated.