This is driving me nuts

M

mitron

New Member
May 24, 2018
13
0
1
44
This is driving me nuts. Spent the last two days trying so many config changes yet having no luck so far.

what I want to archive. Internet should be accessible from every guest.

1) proxmox host has a public ip (x.x.x.x)

2) first guest vm has a public ip (y.y.y.y)

3) second guest vm has a private ip (10.10.10.12)
4) third guest vm has a private ip (10.10.10.13)


The configs I have tried have only partialy worked. Either internet is working in the y.y.y.y guest or the 10.10.0.x range guests.

Havent got a config that makes internet work from all both the public and private guests.

Someone with more experience please share your thoughts
 
could you provide your network configuration (on the Proxmox host it's the file /etc/network/interfaces) and from the public guest and one of the private guests?
if possible both partially working versions
 
VM PUBLIC INTERFACE
Code: 
Contents of the file: /etc/sysconfig/network-scripts/ifcfg-ens18
DEVICE=ens18
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
PEERDNS=no
IPV6INIT=no
IPADDR=94.23.153.81
GATEWAY=188.164.246.254
NETMASK=255.255.255.0
DNS1=213.186.33.99
ARP=yes
HWADDR=02:00:00:DA:xx:xx


VM LOCAL INTERFACE
Code: 
Contents of the file: /etc/sysconfig/network-scripts/ifcfg-ens19
DEVICE=ens19
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
PEERDNS=no
IPV6INIT=no
IPADDR=10.10.10.12
GATEWAY=10.10.10.1
NETMASK=255.255.255.0
DNS1=213.186.33.99
ARP=yes



This version only vmbr1 - can access internet
vmbr0 not working
vmbr1 works - ping to internet and 10.10.10.x ok
///////////////////////////////////////////////////
Code: 
auto lo
iface lo inet loopback

#
# external interface of the host
auto eno1
iface eno1 inet static
        address 188.164.246.137
        netmask 255.255.255.255
        gateway 188.165.246.254

#
# bridge for VMs with public IPs (DMZ)
auto vmbr0
iface vmbr0 inet static
        address 188.164.246.137
        netmask 255.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_fd 0

#
# bridge for internal LAN with private IPs
auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
      
        post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE



This version only vmbr0 - can access internet
vmbr0 works
vmbr1 partly works ping private only - cannot ping internet
///////////////////////////////////////////////////
Code: 
auto lo
iface lo inet loopback

#
# external interface of the host
auto eno1
iface eth0 inet manual

#
# bridge for VMs with public IPs (DMZ)
auto vmbr0
iface vmbr0 inet static
    address 188.164.246.137
    netmask 255.255.255.255
    gateway 188.165.246.254
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0

#
# bridge for internal LAN with private IPs
auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

        post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
 
A few observations:
  • in the "vmbr0 not working" example - you assign the same IP to both eno1 and vmbr0 - this cannot work (an ip only can be assigned to one interface) - and vmbr0 does not have any ports assigned (bridges are like a switch where you "plug networkinterfaces in") - hence the public vm cannot reach anything
  • in the second example you probably have a typo (eno1 vs. eth0) - try to change the post-up/post-down commands to send the packets out of vmbr0 - that should enable the private vms to ping the internet
  • curious about the public vms network config (ifcfg-ens18) since the gatway is not in the same subnet as the ip (and no other routes are provided) - however I'm not really a redhat person - maybe I'm missing something there
 
mitron said:
1) proxmox host has a public ip (x.x.x.x)
2) first guest vm has a public ip (y.y.y.y)
3) second guest vm has a private ip (10.10.10.12)
4) third guest vm has a private ip (10.10.10.13)
Click to expand...

Let me ask you something: Why are you not using the first VM as router/firewall/nat for the rest of your VMs with private IP? There are a few very good reasons to off-load this task from Proxmox-host to one of VMs...
 
@Stoiko

I corrected the eno1/eth0 typo to eno1.
The post-up/post-down commands has been set to send via vmbr0.

Unfortunately it does not work as expected (that should enable the private vms to ping the internet)
From vm with local ip it fails to connect to internet

[root@localhost ~]# ping 216.58.208.238
PING 216.58.208.238 (216.58.208.238) 56(84) bytes of data.
From 10.10.10.1 icmp_seq=1 Destination Host Unreachable
From 10.10.10.1 icmp_seq=2 Destination Host Unreachable
From 10.10.10.1 icmp_seq=3 Destination Host Unreachable
From 10.10.10.1 icmp_seq=4 Destination Host Unreachable
^C
--- 216.58.208.238 ping statistics ---
6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5001ms
 
  • could you please post the output of 'ip route show' and 'ip addr show' (on the host and on the guest)
  • is the host itself able to connect to the internet (ping 8.8.8.8 working)?
 
HOST

Could the issue be on the host there is no route from 10.10.10.1 to internet on the host?
Interent accessible on host and vmbr0 fine.

Code: 
$ ip route show
default via 188.164.246.254  dev eno1 proto static
default via 188.164.246.254  dev vmbr0 proto static
10.10.10.0/24 dev vmbr1 proto kernel scope link src 10.10.10.1
188.164.246.0/24 dev eno1 proto kernel scope link src 188.164.246.137
188.164.246.0/24 dev vmbr0 proto kernel scope link src 188.164.246.137

-------------------------------------------

Code: 
$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 0c:c4:5a:45:35:f6 brd ff:ff:ff:ff:ff:ff
    inet 188.164.246.137/24 brd 188.164.246.255 scope global eno1
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:45:35:f7 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:5a:45:35:f6 brd ff:ff:ff:ff:ff:ff
    inet 188.164.246.137/24 brd 188.164.246.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet 188.164.246.137/32 brd 188.164.246.137 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 2001:41d0:2:c189::/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe45:35f6/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 2e:dc:79:96:62:35 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 brd 10.10.10.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::2cdc:79ff:fe96:6235/64 scope link
       valid_lft forever preferred_lft forever

-------------------------------------------

Code: 
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=4.01 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=4.02 ms


Internet is fine on HOST and vmbr0


=====================================================

VM attached to vmbr1 (NOT working)

Code: 
$ ip route show
default via 10.10.10.1 dev ens18 proto static metric 100
10.10.10.0/24 dev ens18 proto kernel scope link src 10.10.10.10 metric 100

-------------------------------------------

Code: 
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 32:7e:5b:65:bf:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/24 brd 10.10.10.255 scope global noprefixroute ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::307e:5bff:fe65:bfd8/64 scope link
       valid_lft forever preferred_lft forever

-------------------------------------------

Code: 
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 10.10.10.1 icmp_seq=1 Destination Host Unreachable
From 10.10.10.1 icmp_seq=2 Destination Host Unreachable



=====================================================

VM attached to vmbr0 (working)

Code: 
$ ip route show
default via 188.164.246.254  dev ens18 proto static metric 100
94.23.152.0/24 dev ens18 proto kernel scope link src 94.23.152.80 metric 100
188.164.246.254  dev ens18 proto static scope link metric 100

-------------------------------------------

Code: 
$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:00:dd:ed:22 brd ff:ff:ff:ff:ff:ff
    inet 94.23.152.80/24 brd 94.23.152.255 scope global noprefixroute ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:feda:ed21/64 scope link
       valid_lft forever preferred_lft forever

-------------------------------------------

Code: 
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=4.08 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=4.11 ms
 
could I also ask you for the output of 'iptables -nvL -t nat' and 'brctl show'?

A colleague pointed out that with certain hosting providers (afair OVH/Hetzner) the setup of assigning the same IP to multiple interfaces (with different subnetmask) is a valid and working config - however given that your public vm can ping with its own MAC-address - i guess that you don't have this particular setup.

my guess what should work:
  • assign the public ip of the host to vmbr0
  • add eno1 as bridge_port to vmbr0
  • do not assign the same (or another) ip directly to eno1
  • assign 10.0.0.1/24 to vmbr1
  • add the post-up/post-down lines with iptables to vmbr1:
Code: 
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
 
[
  • assign the public ip of the host to vmbr0
  • add eno1 as bridge_port to vmbr0
  • do not assign the same (or another) ip directly to eno1
  • assign 10.0.0.1/24 to vmbr1
  • add the post-up/post-down lines with iptables to vmbr1:
/QUOTE]

I am on OVH and using the approved MAC on host and vm of vmbr0. That bit is working anyway.
Its the NAT through vmbr1 that's not working.
I am already doing the list from your recommendations.

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
#
# bridge for VMs with public IPs (DMZ)
auto vmbr0
iface vmbr0 inet static
address 188.164.246.137
netmask 255.255.255.255
gateway 188.164.246.254
bridge_ports eno1
bridge_stp off
bridge_fd 0


#
# bridge for internal LAN with private IPs
auto vmbr1
iface vmbr1 inet static
address 10.10.10.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward

post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
Click to expand...
 
the config you just pasted does not reflect the output you posted before ( ip route show on the host has a route bound on eno1, both routes on eno1 and vmbr0 have a /24 as netmask (which seems reasonable), the /etc/network/interface you just posted has /32)
changes in the /etc/network/interfaces file do not get applied automatically - you need to restart the networking service (or the host) - but make sure to be able to access the host out of band as to not lose connectivity
if you're on OVH - maybe the documentation can provide you with some help:
https://pve.proxmox.com/wiki/OVH
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back