Networking seems convoluted

[cshill]

Hi Everyone,
Hopefully people here can straighten out some questions I have regarding the networking.

Here's the Bridge Code

auto vmbr0
iface vmbr0 inet static
        address 192.168.XXX.XXX/21
        gateway 192.168.XXX.XXX
        bridge-ports enp4s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Simple Zone Vnet - With the bridge code above I am able to setup a simple zone with a vnet with snat for the subnet I create. The IP is correctly in the range but the subnet is not separated off the main LAN network. It is capable of communicating with the systems on the main LAN which sometimes is ok but what controls do I have over this?

Do I plug an additional NIC card in and bridge one port for this specific subnet of computers and then go to the managed switch and put the port it's plugged into, onto a VLAN?

How about other Proxmox nodes? Can I join other Proxmox nodes into a singular subnetted network without the Proxmox nodes being clustered? Kinda weird question as I would cluster but I have to see if it's possible.



I have setup 3 VLANS all with bridge vmbr0

VNETs
VLAN1 / Zone VLAN1 / Tag 66
VLAN2 / Zone VLAN2 / Tag 77 / VLAN Aware
VLAN3 / Zone VLAN3 / Tag 88 / No VLAN AWARE / Subnet 10.10.10.1


Results:
VLAN 1 doesn't allow you to tag the network device on the VM as the VNET is not VLAN Aware. This is the ultimate "On an Island" experience a VLAN can receive. 0 connections. Seems the only way to give a user access to this is by limiting their permissions and providing the GUI IP for them to access VMs in this setup.

VLAN2 allows tagging the network device of the VM as the VNET and the bridge are VM Aware. This now creates a "Cluster of Islands" where you can have VM's communicate to one another as long as they are on the VLAN2/VNET combo with the VLAN Tag of 77. They however have no gateway so no internet and that's ok. This test is for communication between multiple VMs to confirm functionality of software. Is there a way to have this setup for test VMs where they can't communicate out but developers can remote in? Like maybe I set a random IP to this cluster with a port forward option? Forward 170.25.44.190:8190 - Example - to VM 1. Then 170.25.44.190:8191 is for VM2?

VLAN3 does not allow VLAN Aware with a subnet. Also you can't switch network devices over to this VLAN from VLAN1/VLAN2 due to hotplug problem. Also it does not put the VM onto a subnet even when I have SNAT on. Only time subnetting works is with simple setup, not vlan. So what is the difference with what I want out of this subnetted VLAN and what I'd have to do with the Simple Zone Vnet? The point of subnetting is to expand a LAN or separate off networks so that they can't communicate with one another. I may be wrong but perhaps this VLAN Zone with subnet works when the port from the server is bridged directly to this type of network then the port from the server is plugged into a managed switch on this VLAN?

Desired Outcomes-
I need to have a way of segmenting off VMs in several variations.

1. Segmented off like VLAN1 results - Solved.
2. Segmented off with gateway but limited connection via an approved IP. IE the VM goes directly to the gateway and ignores everything on the LAN except for a single RDP exception from one computer.
3. Segmented off with no gateway but accessible on the LAN. IE, accepted IP addresses IN but no communication OUT of the VM. Like I can copy files in but no one can send ANYTHING out from the VM. If a computer is on my regular LAN I can create a ton of firewall rules so I can do the same on the VM but it has to be sent through Proxmox. This is to test VM clusters for software that won't reach out of the network onto the LAN.
4. Regular Subnet with every VM accessing the internet and communicating with their respective subnet. No subnet leakage like I'm seeing with the Simple Zone.
5. Subnet that CAN communicate with other subnets. How do I configure it so that 10.10.10.1 can communicate with 9.9.9.1 but not 8.8.8.1.

 

[cshill]

Addendum

How do I make multiple bridges to use the same gateway? I have additional NICs, some are going to be on the same LAN as everything else, some are going to be on a separate subnet.

If I have 192.168.40.XXX and 192.168.50.XXX they go out the same gateway while not communicating with each other, but because vmbr0 has that gateway locked down the other NIC can't use it.

 

[cshill]

The solution I had to work with was setup a pfsense firewall/router and use the configuration within there to create 10.10.10.0/24 LAN in order to split up a network. Then a cable for that port into the back of the Proxmox server. The ethernet port on the proxmox associated with 10.10.10.0/24 is set to a virtual bridge with no configuration on it. The information is directly passed from the VMs to the PFsense router/firewall.