How to: bond+bridge+vlan in Proxmox 4.x and share a VLAN with your VM guests

Discussion in 'Proxmox VE: Networking and Firewall' started by x307, May 22, 2016.

  1. x307

    x307 New Member

    Joined:
    May 22, 2016
    Messages:
    11
    Likes Received:
    9
    This configuration (perhaps you'll call it a work-around) took me a while to sort out, so hopefully it will save you some time.

    The problem:
    When you bond 2 interfaces and then want to make a "vmbr" (for example vmbr0) over them, you'll find that the moment a VM starts with the same vlan tag as your proxmox machine, bad things will happen. The VM will not start, or your proxmox machine will loose connection. This is because Linux bridges don't play nice with VLANs.

    Newer versions of the Linux kernel support a "bridge-vlan-aware yes" option, which allows the bridge to pass VLANs properly. This means you can have one vmbr0 bridge for all your VMs to share, and still be able to specify the "vlan tag" in the proxmox GUI. In my experience this works great until a VM is started using the same tag as your proxmox machine (in my case that is VLAN 5), as described above. The workaround I've come up with is to specify a sub-interface with a high offset, such as "vmbr0.5:256". In case it's not obvious to some, this translates to "use vmbr0, on vlan 5, on sub-interface identified by 256).

    Note: the idea of sharing a VLAN for proxmox with your VM guests may not be the best idea for everyone. If you can have your hypervisors on the their own VLAN, then you won't encounter the issue I have described here.

    Hopefully someone will find this useful! If you do, please let me know :D


    root@kvm1:~# cat /etc/network/interfaces

    # network interface settings; autogenerated
    # Please do NOT modify this file directly, unless you know what
    # you're doing.
    #
    # If you want to manage part of the network configuration manually,
    # please utilize the 'source' or 'source-directory' directives to do
    # so.
    # PVE will preserve these directives, but will NOT its network
    # configuration from sourced files, so do not attempt to move any of
    # the PVE managed interfaces into external files!

    auto lo
    iface lo inet loopback

    iface eth0 inet manual

    iface eth1 inet manual

    iface eth2 inet manual

    iface eth3 inet manual

    iface eth91 inet manual

    iface eth92 inet manual

    auto bond0
    iface bond0 inet manual
    slaves eth0 eth1 eth2 eth3
    bond_miimon 100
    bond_mode 802.3ad
    bond_xmit_hash_policy layer2+3​

    auto vmbr0
    iface vmbr0 inet manual
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
    bridge-vlan-aware yes​

    auto vmbr0.5:256
    iface vmbr0.5:256 inet static
    address 10.1.5.101
    netmask 255.255.255.0
    gateway 10.1.5.1
     
    #1 x307, May 22, 2016
    Last edited: May 22, 2016
  2. whiterabbit

    whiterabbit Member
    Proxmox Subscriber

    Joined:
    Dec 19, 2012
    Messages:
    221
    Likes Received:
    4
    Hi. Maybe I've got the same problem ... but can you describe what you mean by
    "until a VM is started using the same tag as your proxmox machine"
     
  3. x307

    x307 New Member

    Joined:
    May 22, 2016
    Messages:
    11
    Likes Received:
    9
    What I mean is: in a scenario where your proxmox node is using the same vlan tag as a VM. Once the VM boots up, it seems to "steal" the vlan from the proxmox host. Using the method above with the predefined sub-interface (vmbr0.5:256) you can avoid this.
     
  4. whiterabbit

    whiterabbit Member
    Proxmox Subscriber

    Joined:
    Dec 19, 2012
    Messages:
    221
    Likes Received:
    4
    Ok, good to know -- but then this is not the scenario we use.
     
  5. valeech

    valeech Member

    Joined:
    May 4, 2016
    Messages:
    39
    Likes Received:
    5
    Matt,

    Thank you so much for this post! I ran into this issue recently and this was exactly the fix I needed.
     
    x307 likes this.
  6. 42n4

    42n4 New Member

    Joined:
    Jun 28, 2016
    Messages:
    3
    Likes Received:
    1
    Yes, it is really a good clue for VLANs in vlan aware bridges. There is no documentation about it in the proxmox wiki. Thank you very much!
    It is the very good setting for virtual proxmox machines (proxmox vms in proxmox) for ceph monitors, they are on the same vlan.
     
    #6 42n4, Feb 2, 2017
    Last edited: Feb 2, 2017
    x307 likes this.
  7. x307

    x307 New Member

    Joined:
    May 22, 2016
    Messages:
    11
    Likes Received:
    9
    Glad this helped you guys :)
     
  8. joelserrano

    joelserrano Member

    Joined:
    Mar 16, 2011
    Messages:
    33
    Likes Received:
    0
    Just wanted to say thanks for sharing! I just ran into the same issue..
     
  9. gdi2k

    gdi2k Member

    Joined:
    Aug 13, 2016
    Messages:
    78
    Likes Received:
    1
    I'm also trying to get this working, but I can't get the Proxmox host talking to the VMs on the VLAN (the VMs can talk to each other on the same VLAN just fine though).

    In my case, I want to have the proxmox server on a Management VLAN with VLAN ID 6. On the server I have this:
    Code:
    auto lo
    iface lo inet loopback
    
    iface enp4s0 inet manual
    
    iface enp3s0 inet manual
    
    iface enp1s0 inet manual
    
    iface enp1s0d1 inet manual
    
    auto bond0
    iface bond0 inet manual
            slaves enp1s0 enp1s0d1
            bond_miimon 100
            bond_mode 802.3ad
            bond_xmit_hash_policy layer2
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.32.0.4
            netmask  255.255.240.0
            gateway  10.32.0.20
            bridge_ports bond0
            bridge_stp off
            bridge_fd 0
            bridge_vlan_aware yes
    
    
    auto vmbr0.6:256
    iface vmbr0.6:256 inet static
    address 10.32.24.10
    netmask 255.255.248.0
    gateway 10.32.24.20
    
    On the Guests I have:

    Code:
    # interfaces(5) file used by ifup(8) and ifdown(8)
    auto lo
    iface lo inet loopback
    
    auto ens18
    iface ens18 inet static
    address  10.32.0.70
    netmask  255.255.240.0
    gateway  10.32.0.20
    
    
    auto ens18.6
    iface ens18.6 inet static
    address  10.32.24.70
    netmask  255.255.240.0
    gateway  10.32.24.20
    
    Not sure where I'm going wrong. Any ideas?
     
  10. whiterabbit

    whiterabbit Member
    Proxmox Subscriber

    Joined:
    Dec 19, 2012
    Messages:
    221
    Likes Received:
    4
    Are you sure about this?
     
  11. gdi2k

    gdi2k Member

    Joined:
    Aug 13, 2016
    Messages:
    78
    Likes Received:
    1
    bond_xmit_hash_policy just sets how the slave device is selected in a LACP link as far as I know. I changed it to layer2+3 as in the OP's first post just in case, but it makes no difference.
     
  12. x307

    x307 New Member

    Joined:
    May 22, 2016
    Messages:
    11
    Likes Received:
    9
    Are you using Proxmox 4 or 5?

    Also, are you specifying the VLAN tag in the Proxmox GUI for each guest? If you are, then you should not have the ".6" on the end of each interface (on the guests)
     
  13. gdi2k

    gdi2k Member

    Joined:
    Aug 13, 2016
    Messages:
    78
    Likes Received:
    1
    This is on Proxmox 5.

    I wasn't specifying the VLAN tag on the Proxmox GUI for the VMs, but I have just tried that too (after disabling VLAN stuff on the VM). The result is the same - I can ping other VMs on VLAN 6, but not the Proxmox host (and vice versa).
     
  14. x307

    x307 New Member

    Joined:
    May 22, 2016
    Messages:
    11
    Likes Received:
    9
    Ok, for Proxmox 5 please check my other post:
    https://forum.proxmox.com/threads/vlan-tag.38051/#post-187740
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice