Help with VM isolation via VLANs

[Venumo]

I'm trying to setup networking to where I can just tag a vm with a vlan so that it can only communicate out to the internet and to other VM's inside that vlan. These VM's are in a private network as well, so I know some form of masquerading will be necessary.
I first tried using a vlan aware bridge, but a tagged vm can't access the gateway and therefore can't reach the internet. So I've started looking into openvswitch however I can't figure it out. Any advice? (And I have read this post: https://forum.proxmox.com/threads/isolating-virtual-machines-with-vlan.92797/ but using firewall rules based on ip won't work for my situation without a lot of extra work)

More detail on my situation if it helps:
I'm currently working on a project for my school where cyber security labs are to be created automatically inside Proxmox, these labs can be 1 to 3 VMs each. I've already got automation using the API going pretty well, my only task I'm stuck on is keeping labs isolated as they can be running malicious code (the rest of the network at our campus is already isolated). I also have no access to a physical switch, it is under lock and key and getting any changes made takes months here.

Any help would be appreciated thanks.

 

[bobmc]

If you create a 'virtual network' for your vm's inside proxmox - they will be able to communicate with each other easily and without needing VLAN's. The trick is then allowing the VM's to reach the internet without reaching elsewhere and using VLAN's alone would not achieve this in any case.

If we assume your 'real' LAN is using 172.16.100.0/24 and your proxmox host is on 172.16.100.50 - for example your network config might look like this

iface lo inet loopback

auto ens18
iface ens18 inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.16.100.50/24
        gateway 172.16.100.1
        bridge-ports ens18
        bridge-stp off
        bridge-fd 0

This gives both your proxmox host and any VM's attached to vmbr0 access to your 'real' LAN

If you add the following
auto vmbr1 iface vmbr1 inet static address 192.168.10.1/24 bridge-ports none bridge-stp off bridge-fd 0

Note that vmbr1 is not connected to any physical network port (bridge-ports none) so the VM's attached to vmbr1 can talk to each other but cannot reach out to the internet - so you have your isolated network.

So that VM's on vmbr1 can reach the internet, there will need to be some form of network routing. Either via VM with two network interfaces, one on vmbr0 and one on vmbr1 (something like pfsense or opnsense would do), or some iptables rules in your network config - e.g
post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 192.168.10.0/24 -o vmbr0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

If you're using iptables, I'm not sure how you would go about preventing the VM's reaching your main LAN as they need to use that to reach the internet - in that case, you may find it easier to connect your vm's to the internet using a firewall VM such as pfSense as you will be able to define access rules using a GUI

HTH

 

[Venumo]

If you create a 'virtual network' for your vm's inside proxmox - they will be able to communicate with each other easily and without needing VLAN's. The trick is then allowing the VM's to reach the internet without reaching elsewhere and using VLAN's alone would not achieve this in any case.

If we assume your 'real' LAN is using 172.16.100.0/24 and your proxmox host is on 172.16.100.50 - for example your network config might look like this

iface lo inet loopback

auto ens18
iface ens18 inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.16.100.50/24
        gateway 172.16.100.1
        bridge-ports ens18
        bridge-stp off
        bridge-fd 0

This gives both your proxmox host and any VM's attached to vmbr0 access to your 'real' LAN

If you add the following
auto vmbr1 iface vmbr1 inet static address 192.168.10.1/24 bridge-ports none bridge-stp off bridge-fd 0

Note that vmbr1 is not connected to any physical network port (bridge-ports none) so the VM's attached to vmbr1 can talk to each other but cannot reach out to the internet - so you have your isolated network.

So that VM's on vmbr1 can reach the internet, there will need to be some form of network routing. Either via VM with two network interfaces, one on vmbr0 and one on vmbr1 (something like pfsense or opnsense would do), or some iptables rules in your network config - e.g
post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 192.168.10.0/24 -o vmbr0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

If you're using iptables, I'm not sure how you would go about preventing the VM's reaching your main LAN as they need to use that to reach the internet - in that case, you may find it easier to connect your vm's to the internet using a firewall VM such as pfSense as you will be able to define access rules using a GUI

HTH

Thank you for your reply, but sorry I should have explained better. I had already basically gotten to this step, my issue is with keeping groups of VM's isolated from one another. I'm spawning VM's from templates dynamically and was hoping I could do something easy like just set the vlan tag through the api. However, when I do this it can't even ping the bridge because it's tagged as a different vlan. To accomplish that I did have to set the vlan aware option on the bridge, so I have tried that.
Is there any way to have translation like NAT but for vlans?


My current fallback idea I've come up with is to just predefine a 50 to 100 bridges with iptable masquerading like your reply, then assign groups of VM's to each bridge and manage which are in use with a database. But this seems dirty and that there should be a better way.

Also, I'm not worried about VM's being able reaching my main LAN, the server I'm working on is inside what is almost a DMZ.