1 pub IP, many VM, NAT doesn't work

[Norrodar]

Hello all together,

I've a very simple problem, but I can't solve it by myself. Even with lots of researches and using the old wiki-page about network-configurations (via web archiver).

However, I got one public IPv4 address and I like to use some VMs using NAT.
So I installed Proxmox per ISO. It autoconfigured ens3 and vmbr0.
I created a new bridge for NATing around, vmbr2.

My config-file looks like this:

#nano /etc/network/interfaces

auto lo
iface lo inet loopback

iface ens3 inet manual

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.xxx  #static public ip
        netmask 255.255.252.0
        gateway xxx.xxx.yyy.yyy # fitting Gateway
        bridge_ports ens3
        bridge_stp off
        bridge_fd 0

auto vmbr2
#private sub network
iface vmbr2 inet static
        address  10.9.0.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.9.0.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.9.0.0/24' -o vmbr0 -j MASQUERADE

I created some Debian 9.6 VMs with 10.9.0.x/24 addresses, using the gateway 10.9.0.1 and the vmbr2 as network-interface (called ens18).
The VMs can't reach the internet. They can't ping eachother and they also can't reach the host 10.9.0.1.

I don't know whats wrong. I tried lots of tutorials, guides, informations from the web. Now I can't help myself anymore.

 

[jim.bond.9862]

Your vmbr2 is an internal bridge. Only used between VMs.
NAT works out of the box on bridge 0
If you want connect VMs to both bridges and use bridge 0 as gateway.
This way vm will use internal network bridge2 to connect to each other and external to go to the net.

 

[Norrodar]

Thank you for your reply.
I don't get it. Shouldn't my postrouting in iptables do the job, translating from vmbr2 to vmbr0 to access the internet?
Additionally if I would do so like you say I'm wondering which IPs I should give the VMs under their second interface vmbr0.
Also shouldn't a VM be able to ping 10.9.0.1 on vmbr2?

I'm really confused right now

 

[jim.bond.9862]

There is nothing to ping.
Your vmbr2 is on different subnet
255.255.255.0
As opposed of your host which is on
255.255.252.0
I am sure you can ping it from host in.
Since I do not know your other settings I can not tell you anything else. Also I am not too strong in iptable routing. I know that I usually do not do this for my home server.
All routing is done by my pfsense box.
The most complicated setup I have is my pfsense box as I run it as vm guest on a dedicated proxmox install.
On that machine I have 3 nice.
I have vmbr0 on one nic. Using it as LAN interface I have local internal o.o on second nic for proxmox access and management. And third is on vmbr1 for WAN
Wan nic is going to my cable modem.
Lan and management to my main switch. Pfsense vm is connected to vmbr0 as lan and vmbr1 as wan. And all routing is done inside. Now if I would use this host for anything else i would simply add the vmbr0 as vm nic and vm would have internet. And network access. The gateway would be my pfsense dhcp.
Unless You are using your host as router and/or firewall nothing special is needed for local nat it is a default setting.

 

[Norrodar]

Hm, I guess that I have a wrong imagination about bridges. I'm looking for something like this (sorry, I can't post images as new user):
h_ttps://i.postimg.cc/SskzwwMH/funny-network.jpg
I used this guide:
h_ttps://raymii.org/s/tutorials/Proxmox_VE_One_Public_IP.html

That's why I'm speaking about pinging between the VMs.
However, I also tried your setup for the same reason: let pfsense handle all that freaking iptables-things. Something like that:
h_ttps://i.postimg.cc/Px55bhZf/pfsense.jpg

For this setup (pfsense as a VM, all other VMs behind it) I got two problems which I couldn't solve:
Both interfaces got a watchdog-timeout (spamming the cli every 10 seconds) and additionally I didn't know how to reach the webgui from outside. I tried to route the LAN-Interface with iptables to an specific port on the WAN-Interface. That was the main problem, to reach the webgui.

 

[Norrodar]

Unless You are using your host as router and/or firewall nothing special is needed for local nat it is a default setting.

Ok, let's say that ens3 and vmbr0 keep its default settings.
ens3 has no entries. vmbr0 has the public/external IP, Gateway and Subnetmask.
If I'm create a VM I'll attach vmbr0 to it. The VM only can get the public IP-Address. I couldn't add another VM to this bridge. Could I? If so I would have to order an additional IP for each VM.
This would be the IPv6 way.

 

[Norrodar]

A little update
I tested around with the proxmox network config and with VMs using Debian 8.0, Debian 9.5, Ubuntu 18.04.
Even if I'm using the config from the wiki the VMs can't connect. If I'm pinging google or the gateway itself: always unreachable.

Additionally I tested some templates (debian, ubuntu) and this containers CAN connect to the internet and they can ping each other using my config from the starting post (using vmbr2). While the VMs still CAN NOT.

For now I'm giving up. Hopefully someone know this problem and can share his/her solution here.
If I solved this problem I definitly post it here in the futur!

Have a nice christmas and a great new year!

 

[jim.bond.9862]

Ok, let's say that ens3 and vmbr0 keep its default settings.
ens3 has no entries. vmbr0 has the public/external IP, Gateway and Subnetmask.
If I'm create a VM I'll attach vmbr0 to it. The VM only can get the public IP-Address. I couldn't add another VM to this bridge. Could I? If so I would have to order an additional IP for each VM.
This would be the IPv6 way.

I don't think so, but you need to make sure that you put all clients on the same subnet if I read your config properly you have your host on one subnet and your VMs on different one. So no ponging between them as there is no router to properly interpreat the address and direct it.

Ok, let's say that ens3 and vmbr0 keep its default settings.
ens3 has no entries. vmbr0 has the public/external IP, Gateway and Subnetmask.
If I'm create a VM I'll attach vmbr0 to it. The VM only can get the public IP-Address. I couldn't add another VM to this bridge. Could I? If so I would have to order an additional IP for each VM.
This would be the IPv6 way.

No vm cannot get any ip without dhcp server. So you need to use static ip or setup dhcp on host, not sure if it's possible.
And yes you can connect multiple VMs on vmbr0 but they will need static ips or you need local dhcp server. If I remember correctly, in the past you could not have dhcp on host
And had to used static address in your kind of setup. Essentially you would have 2 address on the host interface/bridge
Like you would have an public ip on your vmbr0 and add a local aliace ip.
Like you would have public address 67.45.67.0 and an alliase in private range like 192.168.0.0 or 10.0.0.0
All your VMs would have ips in the private range so the can ping the host private ip and nat would take care the internet connection.
You would use iptable to route specific ports from public address to local address/port on vm.

 

[Norrodar]

[...] you need to make sure that you put all clients on the same subnet if I read your config properly you have your host on one subnet and your VMs on different one.[...]

I'm not sure if I got you correct. It's true that the subnetmask has to be the same for all clients in the private network and in the public network. But masks can be different. So the public IP is in my case /22 and the private is in a range of /24.

[...]No vm cannot get any ip without dhcp server. So you need to use static ip or setup dhcp on host, not sure if it's possible.
And yes you can connect multiple VMs on vmbr0 but they will need static ips or you need local dhcp server. If I remember correctly, in the past you could not have dhcp on host. And had to used static address in your kind of setup. Essentially you would have 2 address on the host interface/bridge [...]

Correct, lets DHCP decide the IPs for the clients in the private Network or use static ones.
In my case I give static IPs to my VMs, as I told: 10.9.0.x/24, means:
VM1 -> 10.9.0.2, 255.255.255.0, 10.9.0.1
VM2 -> 10.9.0.3, 255.255.255.0, 10.9.0.1

That means VM1 and VM2 should be able to ping eachother.
The Bridge, that means the Proxmox Host has the 10.9.0.1 IP (as "router").

Like you would have an public ip on your vmbr0 and add a local aliace ip.
Like you would have public address 67.45.67.0 and an alliase in private range like 192.168.0.0 or 10.0.0.0
All your VMs would have ips in the private range so the can ping the host private ip and nat would take care the internet connection.
You would use iptable to route specific ports from public address to local address/port on vm.

I think what you're telling is about simple routing. It isn't cheap to rent a whole subnet, so one IP is fine, NAT does it's routing magic. It's already working with containers only, not with VMs.

"Masquerading allows guests having only a private IP address to access the network by using the host IP address for outgoing traffic. Each outgoing packet is rewritten by iptables to appear as originating from the host, and responses are rewritten accordingly to be routed to the original sender."
Source: https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt