Containers in VLAN have no internet access

[murda]

I have 2 public IP addresses on a single NIC, configured on vmbr0:

• 1.1.1.1 (for PVE management)
• 1.1.1.2 (for a private LAN to NAT to in order to reach the internet)

(Those aren't the real IP addresses, just substituted by me).

And a private IP range 10.0.0.0/24 with the gateway on 10.0.0.1

I want to isolate these 2 public IP addresses for security purposes; I was able to reach the PVE web UI through http://1.1.1.2:8006 for instance, which shouldn't be possible.

My first approach was to create firewall rules for 1.1.1.2 to DROP requests reflecting the ports used by Proxmox VE. This works, but I'm not satisfied with this approach because the 2 spaces aren't logically isolated.

My general understanding is that VLANs are used for isolating networks, so I've gone ahead and made the private LAN's bridge device VLAN Aware and assigned VID 20 to a container. Unfortunately (but logically, to be fair) the internet within the container becomes unreachable.

After some trial and error I'm not sure how to proceed further. Is my desired outcome at all possible?

This would be my network configuration:

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
    address 1.1.1.1/26
    gateway <gateway>
    bridge-ports enp0s31f6
    bridge-stp off
    bridge-fd 0
#Only for PVE management

auto ct1
iface ct1 inet static
    address 10.0.0.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#For containers

source /etc/network/interfaces.d/*

And `/etc/network/interfaces.d/ct1`:

auto vmbr0
iface vmbr0 inet static
    address 1.1.1.2/26

auto ct1
iface ct1 inet static
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    ip-forward on

I added this later to see if it'd work, while commenting out vmbr0 in /etc/network/interfaces.d/ct1 above and changing the `-o` flags to vmbr0.20, but it didn't.

auto vmbr0.20
iface vmbr0.20 inet static
    address 1.1.1.2/26

Thanks.

 

[murda]

I forgot to mention I'm also using DHCP within my containers with dnsmasq on the host which I've set up manually (no SDN).

 

[murda]

2 days later and I'm still struggling with this.

However I just managed to get internet access in my container with the following config:

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
    address 1.1.1.1/26
    gateway <gateway>
    bridge-ports enp0s31f6
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#Only for PVE management

auto vmbr0
iface vmbr0 inet static
    address 1.1.1.2/26

auto vmbr0.50
iface vmbr0.50 inet static
    address 10.0.0.1/24
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

source /etc/network/interfaces.d/*

However this requires me to create a Linux VLAN device for each VLAN tag, and I'd rather not do that because I want each guest to not be able to talk to another by default.

How would I achieve arbitrarily tagging VMs/containers and have it "just work" with a VLAN aware bridge without needing to create a new Linux VLAN device for each new VLAN?

 

[murda]

Conversely this doesn't work...

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
    address 1.1.1.1/26
    gateway <gateway>
    bridge-ports enp0s31f6
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
#Only for PVE management

source /etc/network/interfaces.d/*

### sourced from /etc/network/interfaces.d
auto vmbr0
iface vmbr0 inet static
    address 1.1.1.2/26

auto vmbr0
iface vmbr0 inet static
    address 10.0.0.1/24