Any way to isolate VMs on same VLANs/bridge?

E

EdoFede

Member
Nov 10, 2023
49
26
23
Hi,

I'm planning a migration on a new PVEs infrastructure of many VMs.

One customer have a lot (50, growing) of tiny VMs on the same subnet that needs only internet access (in/out).
They asked if we can "isolate" these VMs.

In the "classic" way for every VM we have to:
- create a VLAN (on PVE and switches)
- creating a subnet, rules, 3 IPs (pfsense with HA) on the firewall
- configuring the VM for the new network
A very long activity (somewhere scriptable, somewhere not)...

Is there any way to achieve a "VM isolation" in proxmox VE for a set of VM that are on the same VLAN/bridge?

In simple terms: every VMs on the same VLAN/subnet must talk only with the default gateway and not with other VMs.

Thanks in advance for any solutions or suggestions!

Bye,
Edoardo
 
  • Like
Reactions: david_devops
Security group added to all VM firewalls that drops all packets that are not coming from/targeting the gateway? But of cause no as isolated as running VLANs. Hadn't time to temper around with the new SDN feature but maybe there you find something.
 
Last edited:
Tried today with PVE Firewall, but seems to do nothing in this case.
I've started 3 VMs, one pinging the other two.
Enabled Firewall on one of the "pinged" VM with DROP policy, nothing happens, ping works.
(of course, I've enabled Firewall also on Datacenter page and on the VMnet adapter).

I was hoping it could be done this way but no success.
And that seems right of course, because firewall works on Layer 3 and I have all the VMs on the same net.

Thanks for your suggestion
 
SteveITS said:
I’ve not tried this here but in my experience with other firewalls, if a connection/state is open adding a drop rule will not close it as the allowed open state still exists. IOW end the ping then enable the rule.
Click to expand...
Proxmox VE uses conntrack, to flush it's tables run in a node shell conntrack -F
 
VLANs are handled at L2, so make sure your network outside of Proxmox isn't routing/allowing the traffic.
 
Sorry for the late reply — old thread, I completely missed the notification.

For the record: @Dunuin's suggestion in #2 was the right one, and it does work. My failed test back in 2023 was almost certainly the conntrack issue @SteveITS and @fba just described — I enabled the drop rule while the pings were already running, so the established state kept passing traffic. `conntrack -F` on the node (or just stopping/restarting the flow) makes the difference.

What I ended up deploying: a single datacenter-level security group applied to every VM that has to be isolated. No SDN, no extra VLANs, no reconfiguration on the VM side.

Datacenter aliases + IPSet + group (dumped from `/etc/pve/firewall/cluster.fw` for readability, but everything was created through the GUI):

Code: 
[ALIASES]

CustA-LAN 192.168.200.0/22 # Customer A LAN
CustA-WebServer 192.168.201.210 # Web Server
CustA-GW 192.168.200.253 # Customer A Gateway
CustA-MailServer 192.168.202.1 # Mail server

[IPSET custa_whitelist] # Customer A whitelist for VM isolation

dc/custa-gw # Gateway
dc/custa-mailserver # Mail server
dc/custa-webserver # Web server

...

[IPSET not-rfc1918] # Internet networks

!10.0.0.0/8 # IPv4 Class A
!169.254.0.0/16 # IPv4 Local link
!172.16.0.0/12 # IPv4 Class B
!192.168.0.0/16 # IPv4 Class C

[IPSET rfc1918] # RFC1918 private networks

10.0.0.0/8 # IPv4 Class A
169.254.0.0/16 # IPv4 Local link
172.16.0.0/12 # IPv4 Class B
192.168.0.0/16 # IPv4 Class C

...

[group vm-custa] # Customer A VMs (gateway + internet only)

IN ACCEPT -source +dc/mgmt-lan -dest dc/custa-lan -log nolog # [Customer A] Allow Management LAN > LAN
IN ACCEPT -source dc/custa-gw -dest dc/custa-lan -log nolog # [Customer A] Allow Gateway > LAN
IN ACCEPT -source +dc/custa_whitelist -dest dc/custa-lan -log nolog # [Customer A] Allow Whitelist > LAN
IN DROP -source dc/custa-lan -dest dc/custa-lan -log info # [Customer A] Deny intra-LAN
IN ACCEPT -source +dc/rfc1918 -dest dc/custa-lan -log nolog # [Customer A] Allow Other LANs (VPN) > LAN
IN ACCEPT -source +dc/not-rfc1918 -dest dc/custa-lan -log nolog # [Customer A] Allow Internet > LAN
OUT ACCEPT -source dc/custa-lan -dest +dc/mgmt-lan -log nolog # [Customer A] Allow LAN > Management LAN
OUT ACCEPT -source dc/custa-lan -dest dc/custa-gw -log nolog # [Customer A] Allow LAN > Gateway
OUT ACCEPT -source dc/custa-lan -dest +dc/custa_whitelist -log nolog # [Customer A] Allow LAN > Whitelist
OUT DROP -source dc/custa-lan -dest dc/custa-lan -log info # [Customer A] Deny intra-LAN
OUT ACCEPT -source dc/custa-lan -dest +dc/rfc1918 -log nolog # [Customer A] Allow LAN > Other LANs (VPN)
OUT ACCEPT -source dc/custa-lan -dest +dc/not-rfc1918 -log nolog # [Customer A] Allow LAN > Internet

And then, on each VM that has to be isolated, `/etc/pve/firewall/<vmid>.fw`:
Code: 
[OPTIONS]

policy_in: ACCEPT
policy_out: ACCEPT
enable: 1

[RULES]

GROUP vm-custa # VM isolation

A few notes on why it's built this way:

- Rules are evaluated top-down, first match wins. The two intra-LAN DROP rules sit *after* the gateway/whitelist ACCEPTs, so the exceptions (gateway, mail, web, management LAN) survive while everything else inside `192.168.200.0/22` is dropped in both directions.
- Both IN and OUT are covered on purpose. Dropping only IN is enough on paper, but doing both means an isolated VM can't initiate anything towards a neighbour either, and the `log info` on those two rules makes lateral-movement attempts visible in the VM firewall log.
- `rfc1918` / `not-rfc1918` are the standard IPSets; splitting the last two ACCEPTs that way keeps VPN/other-subnet traffic and plain internet traffic as separate, individually revocable rules.
- Default policies stay ACCEPT on the VM: all the filtering logic lives in the group, so adding a VM to the isolation set is a one-line change and the whitelist is maintained in exactly one place.
- Firewall must be enabled at datacenter level, on the VM, and on the vNIC (the "Firewall" checkbox on the network device) — all three, otherwise nothing is filtered.

Caveat worth repeating for anyone landing here: this is L3 filtering on the bridge, not real L2 isolation. It's not a substitute for separate VLANs if the threat model includes ARP/L2 games, but for "50 tiny VMs that only need the gateway and internet" it's been solid and enormously cheaper to operate than one VLAN per VM.

Bye,
Edoardo
 
You must log in or register to reply here.
Top Bottom
Back