Bridge network - guest no internet access

Discussion in 'Proxmox VE: Networking and Firewall' started by Tectu, Aug 5, 2018.

Tags:
  1. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    Hello folks,

    I'm currently running Proxmox on a host for evaluation as we're looking for an alternative to ESXi/vSphere.
    The host I'm running the tests on is a baremetal server in a datacenter. The machine has two network interfaces with two public IPs (the machine is "directly connected to the internet"). Let's assume that they are 5.5.5.2 and 5.5.5.3. The gateway is 5.5.5.1.

    So far I managed to get Proxmox working with a couple of VMs using only one of the NICs of the host and having everything configured to provide internet access to the guets via routing/NAT as shown in the documentation.
    Now I'd like to pass through the second NIC directly to one of the guests which will act as a virtual router. I thought that I can simply activate the second NIC on the host and "pass it through" via a bridge interface. However, I haven't been successful in giving that virtual router VM internet access so far and I'd like to seek some assistance.

    Here's my Proxmox host network interfaces configuration file (/etc/network/interfaces):
    Code:
    auto lo
    iface lo inet loopback
    
    # Physical interface 0
    auto enp5s0
    iface enp5s0 inet static
            address  5.5.5.2
            netmask  255.255.255.0
            gateway  5.5.5.1
    
    # Virtual interface for NAT
    auto vmbr0
    iface vmbr0 inet static
            address 10.10.10.1
            netmask 255.255.255.0
            bridge_ports none
            bridge_stp off
            bridge_fd 0
    
            post-up echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
            post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
    
    # Physical interface 1
    auto enp6s0
    iface enp6s0 inet static
            address 5.5.5.3
            netmask 255.255.255.0
    
    # Virtual interface for bridging
    auto vmbr1
    iface vmbr1 inet manual
            bridge_ports enp6s0
            bridge_stp off
            bridge_fd 0
    I wanted to post a screenshot of the web interface but the forum doesn't let me as I'm a new user. However, I don't think that it's necessary anyway as the web interface shows pretty much exactly that configuration.

    My guests are all FreeBSD. However, that shouldn't matter as network configuration is "always the same".
    The guests that use vmbr0 have no problems connecting to the internet at all. However, the one VM that uses vmbr1 is unable to connect. It's also unable to ping the gateway (5.5.5.1).
    Here's my FreeBSD network configuration (from /etc/rc.conf):
    Code:
    ifconfig_em0="inet 5.5.5.3/24"
    defaultrouter="5.5.5.1"
    
    em0 is the guets network interface (like eth0 on most Linux distributions).
    According to ifconfig, the network interface on the guest is up and has the correct IP assigned.

    Could anyone help me out here? What am I missing?
     
  2. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
    The physical interface can't have an IP, when you want to run a bridge on it. Give the IP to VM.
     
  3. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    Thank you for your quick reply.

    Well in general that seems obvious to me but I couldn't get it working. How would my configuration look like? Do I just remove the enp6s0 block entirely to make it look like this?
    Code:
    auto lo
    iface lo inet loopback
    
    # Physical interface 0
    auto enp5s0
    iface enp5s0 inet static
            address  94.126.19.23
            netmask  255.255.255.0
            gateway  94.126.19.1
    
    # Virtual interface for routing (with NAT)
    auto vmbr0
    iface vmbr0 inet static
            address 10.10.10.1
            netmask 255.255.255.0
            bridge_ports none
            bridge_stp off
            bridge_fd 0
    
            post-up echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
            post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
    
    # Virtual interface for bridging on enp6s0
    auto vmbr1
    iface vmbr1 inet manual
            bridge_ports enp6s0
            bridge_stp off
            bridge_fd 0
    
    Unfortunately, I still get the same symptoms: VM that gets vmbr1 assigned is unable to ping anything although the interface is up and has the correct IP assigned.

    I drifted off to BSD quite a lot during the past couple of years so the Linux way of configuring things didn't sticking in my head.
     
  4. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
    As you run the server in a datacenter, restrictions to the network access may apply. Check with the hoster, if there isn't some kind of MAC filter in use.
     
  5. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    I contacted the datacenter and their response is that they don't use any MAC filter or any other restrictions on root servers. It's basically an open line to the internet.
    Any other ideas? :/
     
  6. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
  7. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    I'm not sure where you're getting the idea from that I'm running pfSense. The VM/guest is a bare FreeBSD machine.
    Anyway, this shouldn't be relevant as pfSense itself is based on FreeBSD and after all this stuff is pretty similar across all systems.

    After your last post I realized that I never checked whether the NIC is up on the host. I always just checked the guest. According to ip link the second port (enp6s0) is not up. It's also marked as NO-CARRIER. I tried to bring it up manually using ip link enp6s0 up but that didn't help.

    I've contacted the people at the datacenter again and they told me that the root server only has one physical port even though that two public IPs are available.
    I'm not sure whether Proxmox/the underlying Debian has already been configured to cope with that as ip link clearly shows two dedicated interfaces.

    I'd appreciate any kind of help.
     
  8. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
    pfSense, was an assumption. What does the 'ip link' output give you? Are there two interfaces with different MAC addresses? And how does 'ip addr' look like?
     
  9. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    Here's ip addr of the host machine:
    Code:
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:25:90:ad:23:1e brd ff:ff:ff:ff:ff:ff
        inet 5.5.5.2/24 brd 94.126.19.255 scope global enp5s0
           valid_lft forever preferred_lft forever
        inet6 fe80::225:90ff:fead:231e/64 scope link
           valid_lft forever preferred_lft forever
    3: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
        link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
    4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether a2:51:93:0e:de:6d brd ff:ff:ff:ff:ff:ff
        inet 10.10.10.1/24 brd 10.10.10.255 scope global vmbr0
           valid_lft forever preferred_lft forever
        inet6 fe80::d028:1ff:fe3f:93f6/64 scope link
           valid_lft forever preferred_lft forever
    5: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
        link/ether a2:51:93:0e:de:6d brd ff:ff:ff:ff:ff:ff
    
    and here's ip link:
    Code:
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 00:25:90:ad:23:1e brd ff:ff:ff:ff:ff:ff
    3: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr1 state DOWN mode DEFAULT group default qlen 1000
        link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
    6: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
        link/ether 8a:e5:1a:01:c4:3d brd ff:ff:ff:ff:ff:ff
    7: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
        link/ether 12:63:56:46:6e:41 brd ff:ff:ff:ff:ff:ff
    9: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
        link/ether 00:25:90:ad:23:1f brd ff:ff:ff:ff:ff:ff
    10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/ether 1a:7b:ae:64:28:ac brd ff:ff:ff:ff:ff:ff
    
    This is by using the following configuration file (mind the commented-out section):
    Code:
    auto lo
    iface lo inet loopback
    
    # Physical interface 0
    auto enp5s0
    iface enp5s0 inet static
            address  5.5.5.2
            netmask  255.255.255.0
            gateway  5.5.5.1
    
    # Virtual interface for routing (with NAT)
    auto vmbr0
    iface vmbr0 inet static
            address 10.10.10.1
            netmask 255.255.255.0
            bridge_ports none
            bridge_stp off
            bridge_fd 0
    
            post-up echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
            post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp5s0 -j MASQUERADE
    
    # Phyasical interface 1
    iface enp6s0 inet static
            address 5.5.5.3
            netmask 255.255.255.0
    
    # Virtual interface for bridging on enp6s0
    #auto vmbr1
    #iface vmbr1 inet manual
    #       bridge_ports enp6s0
    #       bridge_stp off
    #       bridge_fd 0
    
    So I assume that this has something to do with the fact that the host has physically just one port but still has two public IPs?

    Thank you for your help, I appreciate it a lot!
     
  10. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
    I would guess, that only one cable is connected. ;)

    Try with 'lspci' and grep for 'ethernet' or similar to see what NIC you got.
     
  11. Tectu

    Tectu New Member
    Proxmox VE Subscriber

    Joined:
    Aug 5, 2018
    Messages:
    6
    Likes Received:
    0
    Well yes. As mentioned in one of my previous post the official statement of the datacenter support team is that the server always only has just one port connected even if two public IPs are available/bought.
    It's not the first root server I am renting and I have never heard of this before. Could somebody confirm that this is plausible?

    According to 'lspci' I have two network adapters on that machine:
    Code:
    05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
    06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
    
     
  12. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,512
    Likes Received:
    131
    The bridge will not come up if the interface is not connected (bridge_ports). I see three possible ways.
    • Add the second public IP to the connected interface and use NAT (as you did already).
    • Put both IPs on the router VM (NIC pass through or via bridge) and distribute the networks from there (if VM is down, access only through iKVM)
    • Get the datacenter to connect the second port of the NIC and use a second bridge without IP (use on the VM)
    I would say, one adapter with two ports (possible onboard)
    Code:
    enp5s0 00:25:90:ad:23:1e
    enp6s0 00:25:90:ad:23:1f
    
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice