Bridge won't start because of bridge-ports issues

Discussion in 'Proxmox VE: Networking and Firewall' started by Marathon, Apr 9, 2019.

  1. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    I have a server with my WAN interface for my Proxmox server where I also have routed subnet on for my VM's.

    The issue I run into is that this works for the IP's fine but I have trouble with the bridge-ports as a bridge doesn't start without any but also complains when it doesn't exist of you use "none" twice, so that is why I now use none1 and none2

    Am I overseeing something here ?

    Code:
    auto lo
    iface lo inet loopback
    iface lo inet6 loopback
    
    auto eth0
    iface eth0 inet static
            address  172.16.0.1
            netmask  255.255.255.0
            gateway  172.16.0.254
    
    auto eth1
    iface eth1 inet manual
    
    auto eth2
    iface eth2 inet manual
    
    auto bond0
    iface bond0 inet manual
            bond-slaves eth1 eth2
            bond-miimon 100
            bond-mode balance-rr
    
    auto vmbr11
    iface vmbr11 inet static
            address  10.0.0.1
            netmask  255.255.255.0
            bridge-ports none1
            bridge-stp off
            bridge-fd 0
    
    auto vmbr101
    iface vmbr101 inet manual
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
    
    auto vmbr1001
    iface vmbr1001 inet manual
            bridge-ports none2
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
     
  2. Richard

    Richard Proxmox Staff Member
    Staff Member

    Joined:
    Mar 6, 2015
    Messages:
    663
    Likes Received:
    23

    the /etc/network/interfaces entry for e.g. vmbr11 is correct (apart from "none1", should be none), if the interface is not up after reboot something is not configured properly in your system.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    Hi,

    The same goes for vmbr1001 then, I can remove them and see no errors with ifreload -a (ifupdown2) but my tapinterfaces, or at least my VM's are not reachable after the reload.

    Any idea about that ?

    Thanks so far!

    /edit because of update

    Proxmox adds 'bridge-ports none' to a bridge where you don't set a bridgeport for. And if you add multiple like that, if updown complains about none not there or only can be used once.

    ifreload works when the birdge-ports are not set, otherwise it errors and the tapinterfaces are not attached anymore

    So actually this is a proxmox-bug, just don't add the bridge-ports line at all when you set nothing there!
     
    #3 Marathon, Apr 10, 2019
    Last edited: Apr 10, 2019
  4. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Thanks for report. I ll look at this with ifupdown2 reload next week.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    Great! Looking forward to it!
     
  6. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Hi,
    I'm trying to reproduce here, but I don't see any problem with reload.
    tap/veth interfaces are still in vmbrX after a networking reload (brctl show vmbrX).

    I'm only seeing an error in "ifreload -a"
    error: 'NoneType' object has no attribute '__getitem__'
    error: 'NoneType' object has no attribute '__getitem__'

    but this is not reload to "bridge-port none", this is because we have a patch in ifupdown2 to not handle tap/veth interfaces not defined in bridge, and I think it's missing a check in some part of reload. (and python code think that tap|veth object exist in ifupdown2 cache, and throw an error on this).
    I'll fix that, but I don't think they are any impact.

    Is is this error message that you seen on reload ?


    Edit:
    I'm also seeing a breaking, if a bridge-xxxx is changed, the vms/ct tap/veth interfaces are not in bridge after reload. (taking default bridge-ports ...) . I'll check that.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #6 spirit, Apr 16, 2019
    Last edited: Apr 16, 2019
  7. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    Good we are making some progress, nice you checked this out.

    The error I get is the following:

    # ifreload -a
    error: vmbr11: bridge port none1 does not exist

    After that non of the hosts are pingable anymore and I need to bring down/up the hosts for a working tap again.
     
  8. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Hi,

    could you verify in
    /etc/network/ifupdown2/ifupdown2.conf

    the value of
    ifreload_down_changed.

    and try to setup it to 1 if it's 0.

    ifreload_down_changed=1

    and use "bridge-ports none" in your config. (not none1)


    the thing to check, is after a reload, do a "brctl show", and check if tap interfaces are still in the bridge.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    Hi,

    Thanks so far!

    I made the setting and did a reload but the same issue occures, same error, even on none. I changed that for the setup as I hoped it checked like you can also bridge only once to an interface and none was used multiple times, I now use it only on one bridge

    Code:
    # brctl show
    bridge name     bridge id               STP enabled     interfaces
    vmbr1001                8000.6cb3113c80a6       no              bond0
    vmbr11          8000.000000000000       no
    It now errors on vmbr11 as that one has no bridge interface, so is set to none and vmbr1001 doesn't give a ping anymore as well.

    Does this help ?
     
  10. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    I have build an updated version of ifupdown2, with changes to handle tap|veth interfaces,

    http://odisoweb1.odiso.net/ifupdown2_1.2.5-1+pvetest1_all.deb

    can you test it ?


    Also, about your configuration, do you have now:

    ?

    are can you post your vms configuration ? (/etc/pve/qemu-server/<vmid>.conf)


    Edit: for bridge vlan aware, it seem than vlan tag is lost on ifreload
    (#bridge -c vlan to verify )

    Edit: Ok, I have fixed it, and reuploaded again
    http://odisoweb1.odiso.net/ifupdown2_1.2.5-1+pvetest1_all.deb

    I have tested with/without none, with/without bridge vlan aware, and I don't have breakeage anymore.
    Also, it's really need "ifreload_down_changed=0" in ifupdown2.conf
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #10 spirit, Apr 16, 2019
    Last edited: Apr 17, 2019
    dkorzhevin likes this.
  11. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    Hi,

    I have tested it and it seems to work perfectly now, no breakage here as well anymore, thanks for that!

    The only message I got was:

    Code:
    # ifreload -a
    error: /etc/network/interfaces: line36: iface bond0: unsupported keyword (bond-primary)
    error: /etc/network/interfaces: line37: iface bond0: unsupported keyword (bond-primary-reselect)
    
    I thought that was supported (and works).
     
  12. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Great :)

    patches have been applied in proxmox git and official package should be available soon.

    Thanks again for the report and tests.


    seem than bond-primary is not yet implemented in ifupdwon2
    https://github.com/CumulusNetworks/ifupdown2/issues/9

    here a work around

    Code:
    auto bond0
    iface bond0 inet dhcp
            bond-slaves eth0 wlan0
            bond-mode active-backup
            up echo eth0 > /sys/class/net/$IFACE/bonding/primary
    
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    You thanks for the quick response and late work! Really appreciated, I'm happy with it!

    About the bonds let's hope that in there soon! The workaround is a good idea, thanks!

    Maybe you are able to look into offloading as well on the tap interface ? Something I think you have an idea about:

    https://forum.proxmox.com/threads/tx-rx-offloading-on-tap-interface.53350
     
  14. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    OK, everything works so far but traffic between vmbr11 and eth0 stops floating after a ifreload -a.

    On the proxmox machine itself I can ping to the outside world and the IP's on tap interfaces of vmbr11 and vmbr11 itself but not over eth0 anymore, so form the outside world.

    Maybe I'm missing something but you might have an idea as well ?
     
  15. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Hi

    what do you mean by traffic between eth0 and vmbr11 ? (eth0 is not in bridge, and they are on differents subnets)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    It's a subnet routed on the eth0 port so the eth0 address is the gateway of the subnet.
     
  17. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    can you verify that ip_foward is still enabled after reload ?

    cat /proc/sys/net/ipv4/ip_forward

    ?

    also, do you have any kind of static routes in /etc/network/interfaces ?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. Marathon

    Marathon Member

    Joined:
    Apr 7, 2019
    Messages:
    38
    Likes Received:
    0
    I need to check, not able at the moment, sorry for that and inform you later about it!

    I don't have any static routes in the interfaces file (that is where routers are for ;))
     
  19. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,312
    Likes Received:
    131
    can you send result of:

    grep "" /proc/sys/net/ipv4/conf/*/forwarding


    before and after reload ?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice