Bridge network - guest no internet access

Aug 5, 2018
12
0
1
30
Hello folks,

I'm currently running Proxmox on a host for evaluation as we're looking for an alternative to ESXi/vSphere.
The host I'm running the tests on is a baremetal server in a datacenter. The machine has two network interfaces with two public IPs (the machine is "directly connected to the internet"). Let's assume that they are 5.5.5.2 and 5.5.5.3. The gateway is 5.5.5.1.

So far I managed to get Proxmox working with a couple of VMs using only one of the NICs of the host and having everything configured to provide internet access to the guets via routing/NAT as shown in the documentation.
Now I'd like to pass through the second NIC directly to one of the guests which will act as a virtual router. I thought that I can simply activate the second NIC on the host and "pass it through" via a bridge interface. However, I haven't been successful in giving that virtual router VM internet access so far and I'd like to seek some assistance.

Here's my Proxmox host network interfaces configuration file (/etc/network/interfaces):
Code:
auto lo
iface lo inet loopback

# Physical interface 0
auto enp5s0
iface enp5s0 inet static
        address  5.5.5.2
        netmask  255.255.255.0
        gateway  5.5.5.1

# Virtual interface for NAT
auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE

# Physical interface 1
auto enp6s0
iface enp6s0 inet static
        address 5.5.5.3
        netmask 255.255.255.0

# Virtual interface for bridging
auto vmbr1
iface vmbr1 inet manual
        bridge_ports enp6s0
        bridge_stp off
        bridge_fd 0
I wanted to post a screenshot of the web interface but the forum doesn't let me as I'm a new user. However, I don't think that it's necessary anyway as the web interface shows pretty much exactly that configuration.

My guests are all FreeBSD. However, that shouldn't matter as network configuration is "always the same".
The guests that use vmbr0 have no problems connecting to the internet at all. However, the one VM that uses vmbr1 is unable to connect. It's also unable to ping the gateway (5.5.5.1).
Here's my FreeBSD network configuration (from /etc/rc.conf):
Code:
ifconfig_em0="inet 5.5.5.3/24"
defaultrouter="5.5.5.1"
em0 is the guets network interface (like eth0 on most Linux distributions).
According to ifconfig, the network interface on the guest is up and has the correct IP assigned.

Could anyone help me out here? What am I missing?
 
The physical interface can't have an IP, when you want to run a bridge on it. Give the IP to VM.
 
Thank you for your quick reply.

Well in general that seems obvious to me but I couldn't get it working. How would my configuration look like? Do I just remove the enp6s0 block entirely to make it look like this?
Code:
auto lo
iface lo inet loopback

# Physical interface 0
auto enp5s0
iface enp5s0 inet static
        address  94.126.19.23
        netmask  255.255.255.0
        gateway  94.126.19.1

# Virtual interface for routing (with NAT)
auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE

# Virtual interface for bridging on enp6s0
auto vmbr1
iface vmbr1 inet manual
        bridge_ports enp6s0
        bridge_stp off
        bridge_fd 0
Unfortunately, I still get the same symptoms: VM that gets vmbr1 assigned is unable to ping anything although the interface is up and has the correct IP assigned.

I drifted off to BSD quite a lot during the past couple of years so the Linux way of configuring things didn't sticking in my head.
 
As you run the server in a datacenter, restrictions to the network access may apply. Check with the hoster, if there isn't some kind of MAC filter in use.
 
I contacted the datacenter and their response is that they don't use any MAC filter or any other restrictions on root servers. It's basically an open line to the internet.
Any other ideas? :/
 
I'm not sure where you're getting the idea from that I'm running pfSense. The VM/guest is a bare FreeBSD machine.
Anyway, this shouldn't be relevant as pfSense itself is based on FreeBSD and after all this stuff is pretty similar across all systems.

After your last post I realized that I never checked whether the NIC is up on the host. I always just checked the guest. According to ip link the second port (enp6s0) is not up. It's also marked as NO-CARRIER. I tried to bring it up manually using ip link enp6s0 up but that didn't help.

I've contacted the people at the datacenter again and they told me that the root server only has one physical port even though that two public IPs are available.
I'm not sure whether Proxmox/the underlying Debian has already been configured to cope with that as ip link clearly shows two dedicated interfaces.

I'd appreciate any kind of help.
 
pfSense, was an assumption. What does the 'ip link' output give you? Are there two interfaces with different MAC addresses? And how does 'ip addr' look like?
 
Here's ip addr of the host machine:
Code:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:25:90:ad:23:1e brd ff:ff:ff:ff:ff:ff
    inet 5.5.5.2/24 brd 94.126.19.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fead:231e/64 scope link
       valid_lft forever preferred_lft forever
3: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a2:51:93:0e:de:6d brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 brd 10.10.10.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d028:1ff:fe3f:93f6/64 scope link
       valid_lft forever preferred_lft forever
5: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether a2:51:93:0e:de:6d brd ff:ff:ff:ff:ff:ff
and here's ip link:
Code:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:ad:23:1e brd ff:ff:ff:ff:ff:ff
3: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr1 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
6: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 8a:e5:1a:01:c4:3d brd ff:ff:ff:ff:ff:ff
7: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 12:63:56:46:6e:41 brd ff:ff:ff:ff:ff:ff
9: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1a:7b:ae:64:28:ac brd ff:ff:ff:ff:ff:ff

This is by using the following configuration file (mind the commented-out section):
Code:
auto lo
iface lo inet loopback

# Physical interface 0
auto enp5s0
iface enp5s0 inet static
        address  5.5.5.2
        netmask  255.255.255.0
        gateway  5.5.5.1

# Virtual interface for routing (with NAT)
auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE

# Phyasical interface 1
iface enp6s0 inet static
        address 5.5.5.3
        netmask 255.255.255.0

# Virtual interface for bridging on enp6s0
#auto vmbr1
#iface vmbr1 inet manual
#       bridge_ports enp6s0
#       bridge_stp off
#       bridge_fd 0

So I assume that this has something to do with the fact that the host has physically just one port but still has two public IPs?

Thank you for your help, I appreciate it a lot!
 
So I assume that this has something to do with the fact that the host has physically just one port but still has two public IPs?
I would guess, that only one cable is connected. ;)

Try with 'lspci' and grep for 'ethernet' or similar to see what NIC you got.
 
I would guess, that only one cable is connected. ;)
Well yes. As mentioned in one of my previous post the official statement of the datacenter support team is that the server always only has just one port connected even if two public IPs are available/bought.
It's not the first root server I am renting and I have never heard of this before. Could somebody confirm that this is plausible?

According to 'lspci' I have two network adapters on that machine:
Code:
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
 
Well yes. As mentioned in one of my previous post the official statement of the datacenter support team is that the server always only has just one port connected even if two public IPs are available/bought.
The bridge will not come up if the interface is not connected (bridge_ports). I see three possible ways.
  • Add the second public IP to the connected interface and use NAT (as you did already).
  • Put both IPs on the router VM (NIC pass through or via bridge) and distribute the networks from there (if VM is down, access only through iKVM)
  • Get the datacenter to connect the second port of the NIC and use a second bridge without IP (use on the VM)
According to 'lspci' I have two network adapters on that machine:
I would say, one adapter with two ports (possible onboard)
Code:
enp5s0 00:25:90:ad:23:1e
enp6s0 00:25:90:ad:23:1f
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!