Proxmox in NAT but no internet on VM

Guiguii · May 10, 2023

Hi

I'll make it very simple, I've been stuck on Proxmox for a very long time, I'm desperate.

Currently, I have a network configuration with a physical card in DHCP and a vmbr0 virtual card which is in NAT on enp2s0

I then set up a DHCP server on the Proxmox so that an ip was automatically assigned to a VM on the 192.168.124.X site.

The DHCP connection is fine when installing a VM all you want, I can ping the hypervisor, BUT, I don't don't have internet.

I thought it was a DNS problem at first but obviously it's not.

Please help me...

I am OBLIGED to do NAT, I have absolutely no right to bridge

my file interfaces

Code:

auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet dhcp
    dns-nameservers 192.168.X.X
    post-up ip route add default via 192.168.X.Xdev enp2s0 proto dhcp src 192.168.X.X metric 100

auto vmbr0
iface vmbr0 inet static
    address 192.168.124.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '192.168.124.0/24' -o enp2s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.124.0/24' -o enp2s0 -j MASQUERADE
    post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

One of my theories would be that the packet goes out and reaches the server, but does not come back... Maybe I'm wrong, I'm really not good at it...

Chris · May 10, 2023

Hi,
first you should check if you can ping your default gateway. Please post the output of ip addr; ip route. No need to retract local IP addresses.

Guiguii said:
post-up ip route add default via 192.168.X.Xdev enp2s0 proto dhcp src 192.168.X.X metric 100

Do you need to set the default route? Is this not given by DHCP? Are you sure you have the correct gateway IP here?

Guiguii · May 10, 2023

Hi Chris,
Thanks for you answer,

This is my ip addr

Code:

ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d8:cb:8a:c2:c8:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.X.X/24 brd 192.168.0.255 scope global dynamic enp2s0
       valid_lft 42929sec preferred_lft 42929sec
    inet6 fe80::dacb:8aff:fec2:c836/64 scope link
       valid_lft forever preferred_lft forever
3: (Voluntarily deleted the 3 because useless in our case, it is a "backup" card connected to nothing at all)
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 10.8.0.1/24 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fddd:1194:1194:1194::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::657b:80aa:a657:9b3e/64 scope link stable-privacy
       valid_lft forever preferred_lft forever
14: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 4e:91:1f:ae:6b:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever

And my ip route

Code:

default via 192.168.X.X dev enp2s0
default via 192.168.X.X dev enp2s0 proto dhcp src 192.168.X.X metric 100
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1
192.168.0.0/24 dev enp2s0 proto kernel scope link src 192.168.X.X
192.168.124.0/24 dev vmbr0 proto kernel scope link src 192.168.124.1 linkdown

The DNS is the one provided by my internet service provider.

"Do you need to set the default route? Is this not given by DHCP? Are you sure you have the correct gateway IP here?"
Hmm yeah, probably given by DHCP but i can keep it actually ?

Chris · May 10, 2023

Guiguii said:

And my ip route

Code:

default via 192.168.X.X dev enp2s0
default via 192.168.X.X dev enp2s0 proto dhcp src 192.168.X.X metric 100
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1
192.168.0.0/24 dev enp2s0 proto kernel scope link src 192.168.X.X
192.168.124.0/24 dev vmbr0 proto kernel scope link src 192.168.124.1 linkdown

You have duplicate routing entries for the default gateway. That is why i was asking about the entry in your network config. Please try to remove that and reload the config with ifreload -a

Guiguii said:
14: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

Also, it seems like your bridge is down?

Guiguii · May 10, 2023

Chris said:
You have duplicate routing entries for the default gateway. That is why i was asking about the entry in your network config. Please try to remove that and reload the config with ifreload -a

Also, it seems like your bridge is down?

Deleted!
I can't bridge, I have to keep enp2s0 independent, and keep its ip, it must host a VPN in the future (for example), hence the NAT connection attempt

Chris · May 10, 2023

Guiguii said:
Deleted!
I can't bridge, I have to keep enp2s0 independent, and keep its ip, it must host a VPN in the future (for example), hence the NAT connection attempt

You will still attach the VMs/CTs to the internal bridge and only masquerade when traffic has to actually leave the host trough enp2s0. Your bridge has to be up

Guiguii · May 10, 2023

Chris said:
You will still attach the VMs/CTs to the internal bridge and only masquerade when traffic has to actually leave the host trough enp2s0. Your bridge has to be up

So I just add the line bridge-ports enp2s0 to my config file?

I'm sorry but I've seen and read so much that I'm completely lost...

Like this?

Code:

auto vmbr0
iface vmbr0 inet static
    address 192.168.124.1/24
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '192.168.124.0/24' -o enp2s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.124.0/24' -o enp2s0 -j MASQUERADE
    post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

bobmc · May 10, 2023

Guiguii said:
Deleted!
I can't bridge, I have to keep enp2s0 independent, and keep its ip, it must host a VPN in the future (for example), hence the NAT connection attempt

Having a bridge does not preclude NAT, you only need to have a config that fits your NAT rules.

Both of your networks are in the 192.168.0.0/16 subnet - i.e they are both private networks so the way you are approaching this would mean you would be having to do double-NAT to reach your VM's from an external site, not sure this is what you're intending?

It might be best if you explain what you're trying to achieve so that we can advise the best way of getting there?

Guiguii · May 10, 2023

Ok I will explain what is desired.

I will try to explain with an example; Suppose we have a machine with a network card that we will call eth0, and on it we put, I don't know, virt-manager for example. And well on this server I could make a connection route between eth0 and virbr0 with virsh, so eth0 will have its ip which we will say 192.168.0.28 and virbr0 will have its ip we will say 192.168.121.1.
So we have an eth0 card which is independent, which allows you to put VPNs, and other tools on it (firewall or I don't know), and we have a card which is then dedicated just to VMs.

For Proxmox I try to do the same thing, with enp2s0 which is always independent, so that is to say that it will just have its own ip like 192.168.0.80, and I can do my VPN / firewall type things for example (not necessarily that but that it is not "sucked up" by the virtual card), and vmbr0 with his ip like 192.168.0.122 which will be just there to give internet to my VMs which will be on Proxmox without interfering with enp2s0

I hope I'm clear enough, I'm sorry...

bobmc · May 10, 2023

If I understand you correctly;

a) you want your proxmox host to be on the 'main' network and accessible from there and potentially from outside your network via NAT
b) you want your virtual machines to be on their own network but they should be able to access the internet if required

So a config like this would work in that manner

Code:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.80/24
        gateway 192.168.0.1
        bridge-ports ens18
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.121.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
post-up   iptables -t nat -A POSTROUTING -s 192.168.121.0/24 -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s 192.168.121.0/24 -o vmbr0 -j MASQUERADE
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

In this scenario, the VM's on vmbr1 could reach the internet but there would be no direct access to them from your main network. However, a VM on vmbr0 would be on the main network.

If you wanted to have a VM running say a webserver on vmbr1 then you could NAT onto that VM using additional iptables rules - e.g

Code:

post-up   iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to-destination 192.168.121.101
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to-destination 192.168.121.101

or you could consider running a VM to do network routing between the main lan and the vm lan (e.g pfSense/OpnSense etc) with a 'WAN' interface on vmbr0 and a 'LAN' interface on vmbr1

Hope this helps

Guiguii · May 11, 2023

I have to bridge, can't do the setup any other way?

bobmc said:
If I understand you correctly;

a) you want your proxmox host to be on the 'main' network and accessible from there and potentially from outside your network via NAT
b) you want your virtual machines to be on their own network but they should be able to access the internet if required

So a config like this would work in that manner

Code:

auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.0.80/24 gateway 192.168.0.1 bridge-ports ens18 bridge-stp off bridge-fd 0 auto vmbr1 iface vmbr1 inet static address 192.168.121.1/24 bridge-ports none bridge-stp off bridge-fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.121.0/24 -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s 192.168.121.0/24 -o vmbr0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

In this scenario, the VM's on vmbr1 could reach the internet but there would be no direct access to them from your main network. However, a VM on vmbr0 would be on the main network.

If you wanted to have a VM running say a webserver on vmbr1 then you could NAT onto that VM using additional iptables rules - e.g

Code:

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to-destination 192.168.121.101 post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to-destination 192.168.121.101

or you could consider running a VM to do network routing between the main lan and the vm lan (e.g pfSense/OpnSense etc) with a 'WAN' interface on vmbr0 and a 'LAN' interface on vmbr1

Hope this helps

Thanks for ur help bobmc,

Am I obliged to play bridge? There is no possibility of doing otherwise?

bobmc · May 11, 2023

Guiguii said:
Am I obliged to play bridge? There is no possibility of doing otherwise?

No, you don't have to bridge, but I think you gain more from having a bridge than not. Is there a reason why you don't want to bridge?

Search

Search

Proxmox in NAT but no internet on VM

Guiguii

New Member

Chris

Proxmox Staff Member

Guiguii

New Member

Chris

Proxmox Staff Member

Guiguii

New Member

Chris

Proxmox Staff Member

Guiguii

New Member

bobmc

Renowned Member

Guiguii

New Member

bobmc

Renowned Member

Guiguii

New Member

bobmc

Renowned Member