NAT networking but with multiple bridges?

[rlindsay]

I've set my system up to use NAT. Which works ok, I can give my containers network access and run services on them
However I am currently just running one bridge for the network and extending that with new pre and post iptables routes for each port on each container I want to expose.
The problem is, when you add a new container, you need to bump the bridge to load the new config. Which disconnects all the other containers. Which is a bit of a pain
Is there a more flexible way of doing this.
My thought was to create a bridge with a small subnet range, like a /28 for a bridge. Assign each container a bridge. Then when I want to restart the network for that container I only impact that container.

This is my example
auto enp6s0 iface enp6s0 inet manual auto enp7s0 iface enp7s0 inet manual iface enp8s0 inet manual iface enp9s0 inet manual auto bond0 iface bond0 inet static address 10.126.24.82/24 gateway 10.126.24.1 bond-slaves enp6s0 enp7s0 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 auto vmbr0 iface vmbr0 inet static address 192.168.10.14/28 bridge-ports none bridge-stp off bridge-fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.10.0/28 -o bond0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 192.168.10.0/28 -o bond0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 post-up iptables -t nat -A PREROUTING -p tcp --dport 2212 -j DNAT --to-destination 192.168.10.2:22 post-down iptables -t nat -A PREROUTING -p tcp --dport 2212 -j DNAT --to-destination 192.168.10.2:22 post-up iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.2 -j SNAT --to-source 192.168.10.14:2212 post-down iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.2 -j SNAT --to-source 192.168.10.14:2212 post-up iptables -t nat -A PREROUTING -p tcp --dport 2210 -j DNAT --to-destination 192.168.10.3:22 post-down iptables -t nat -A PREROUTING -p tcp --dport 2210 -j DNAT --to-destination 192.168.10.3:22 post-up iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.3 -j SNAT --to-source 192.168.10.14:2210 post-down iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.3 -j SNAT --to-source 192.168.10.14:2210 post-up iptables -t nat -A PREROUTING -p tcp --dport 8010 -j DNAT --to-destination 192.168.10.3:80 post-down iptables -t nat -A PREROUTING -p tcp --dport 8010 -j DNAT --to-destination 192.168.10.3:80 post-up iptables -t nat -A PREROUTING -p tcp --sport 80 -j DNAT --to-destination 192.168.10.14:8010 post-down iptables -t nat -A PREROUTING -p tcp --sport 80 -j DNAT --to-destination 192.168.10.14:8010 post-up iptables -t nat -A PREROUTING -p tcp --dport 2211 -j DNAT --to-destination 192.168.10.10:22 post-down iptables -t nat -A PREROUTING -p tcp --dport 2211 -j DNAT --to-destination 192.168.10.10:22 post-up iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.10 -j SNAT --to-source 192.168.10.14:2211 post-down iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.10 -j SNAT --to-source 192.168.10.14:2211 auto vmbr1 iface vmbr1 inet static address 192.168.10.16/28 bridge-ports none bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.10.16/28 -o bond0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 192.168.10.16/28 -o bond0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 post-up iptables -t nat -A PREROUTING -p tcp --dport 2214 -j DNAT --to-destination 192.168.10.17:22 post-down iptables -t nat -A PREROUTING -p tcp --dport 2214 -j DNAT --to-destination 192.168.10.17:22 post-up iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.17 -j SNAT --to-source 192.168.10.16:2214 post-down iptables -t nat -A POSTROUTING -p tcp --sport 22 -s 192.168.10.17 -j SNAT --to-source 192.168.10.16:2214

Where I have my bond0 setup.
vmbr0 is the first NATed bridge I created as I mentioned.
Then I have my pre and post routing firewall rules for some of my containers. Which as you can see gets a bit unwieldy
so I created vmbr1 as an additional bridge. This one worked partially. I can bring the bridge up and connect to this from the proxmox command line. However when I tried to ssh to this one, ssh wouldn't make a connection to it.
Dont know if I need to make a new zone . i.e.
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 2
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 2

Any help would be ideal

 

[ph0x]

Creating a firewall VM with vmbr0 as WAN and another bridge as LAN, where all containers live, would probably solve the problem.

 

[rlindsay]

Hmm, don't really see how that helps. I want to basically isolate each container from the others networking.

 

[ph0x]

You could add another bridge on PVE for each container and add it as an interface to the firewall VM.

 

[rlindsay]

You wouldn't have an example of how that works in reality? Do you have to do anything to let each of the bridges talk to each other and route traffic?

 

[ph0x]

Since every bridge is plugged into the firewall VM as a separate network interface you can decide to route traffic between them or not. Standard behaviour is no communication between the networks.
I have to admit that I didn't have the need for that setup so far, so it is merely a suggestion, but I don't see why it shouldn't work. Plus, you don't have the hassle with the NAT on your host.

 

[rlindsay]

So this is really what I have currently.
I'm very limited because essentially I only have 1 external facing IP
I can't do any additional routing beyond the external IP because of the corporate infrastructure.
I would like to be able to connect container C in a similar manner in that I can expose say web ports on it and run services and make it available via the same external IP address.
However I would like to be able to have it's bridge or whatever mechanism work independently of the existing VMBR 0. As if I need to restart the network interfaces for container C I don't also want to bring down the other two containers

I know LXC has macvtap capability. Is that something I can run here in proxomox?

 

[bobmc]

I think you're making life unnecessarily for yourself. Yes you can get down'n'dirty and do things like this with iptables but as @ph0x has already suggested, you should consider installing a firewall/router os like pfSense as a VM on the host and this will allow you to route traffic to any vm's and isolate them where necessary. You can create VLANs and also do ipsec and vpn tunnels to give you remote access as needed.

https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
https://youtu.be/hdoBQNI_Ab8