[SOLVED] bond_mode active-backup issue

Discussion in 'Proxmox VE: Networking and Firewall' started by RobFantini, Oct 10, 2018.

  1. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    Hello

    we are currently using bonds set up like this:
    Code:
    auto bond0
    iface bond0 inet manual
            slaves enp8s0f0 enp8s0f1
            bond_miimon 100
            bond_mode 802.3ad
            bond_xmit_hash_policy layer2+3
    
    those work, except I read for cluster that type of bond should not be used:
    https://pve.proxmox.com/pve-docs/pve-network-plain.html
    If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported.

    When I change over to active-backup [ used pve > network and restarted node] , the network does not work. no ping out etc.
    Code:
    auto bond0
    iface bond1 inet manual
            slaves enp8s0f0 enp8s0f1
            bond_miimon 100
            bond_mode active-backup
    
    We are using a managed switch . the two wires connect to lagged ports. I've tried the 6 different hash types.

    Does anyone have a clue on what to try next?
     
  2. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    Using non lag ports on the 2 switches - network works.

    So for active-backup - lag ports should not be used it seems..
     
  3. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,070
    Likes Received:
    250
    Hi,RobFantini,

    the general recommendation (Docu has to be more precise) is to use a dual ring approach for corosync network and not bonds.
    If you have to use a bond you should use active-backup but the two switches have to be connected.
    LACP would also work but you need MLAG to cross connect the switches and also IGMP has to work correctly.
    The downside of this approach is that the latency will/can rise and this can be a problem.
    All other bond modes have the problem of packages out of order.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    Hello Wolfgang,
    thank you for the response. we had tried using bond before , and ran in to the latency issues. i thought it was a hardware issue.

    please help me see if I got this right:

    So the dual ring network would be used just for cluster communication. as part of adding a node to cluster : network hardware , wires and configuration would get done 1st.

    storage and vm's would use other networks.


    ** could you give and example of a /etc/network/interfaces file dual ring set up?

    PS:
    there always more to learn ! every time i think i know more than 50% the other 50% starts to show up.

    Have you seen a dual ring wire set up on a server rack rack ? will need to figure that out.
     
  5. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,070
    Likes Received:
    250
    Here the configs
    PHP:

    auto lo
    iface lo inet loopback

    iface ens18 inet manual
    # VM network

    auto ens19
    iface ens19 inet 
    static
        
    address  10.10.19.60
        netmask  255.255.255.0
    # ring0

    auto ens20
    iface ens20 inet 
    static
        
    address  10.10.20.60
        netmask  255.255.255.0
    #ring1

    iface ens21 inet manual

    iface ens22 inet manual

    auto bond0
    iface bond0 inet 
    static
        
    address  10.10.10.60
        netmask  255.255.255.0
        bond
    -slaves ens21 ens22
        bond
    -miimon 100
        bond
    -mode 802.3ad
        bond
    -xmit-hash-policy layer3+4
    # storage net

    auto vmbr0
    iface vmbr0 inet 
    static
        
    address  192.168.18.60
        netmask  255.255.240.0
        gateway  192.168.16.1
        bridge
    -ports ens18
        bridge
    -stp off
        bridge
    -fd 0
    PHP:

    logging 
    {
      
    debugoff
      to_syslog
    yes
    }

    nodelist {
      
    node {
        
    namepve0
        nodeid
    1
        quorum_votes
    1
        ring0_addr
    10.10.20.60
        ring1_addr
    10.10.19.60
      
    }
      
    node {
        
    namepve1
        nodeid
    2
        quorum_votes
    1
        ring0_addr
    10.10.20.61
        ring1_addr
    10.10.19.61
      
    }
      
    node {
        
    namepve2
        nodeid
    3
        quorum_votes
    1
        ring0_addr
    10.10.20.62
        ring1_addr
    10.10.19.62 
      
    }
      
    node {
        
    namepve3
        nodeid
    4
        quorum_votes
    1
        ring0_addr
    10.10.20.63
        ring1_addr
    10.10.19.63 
     
    }

    }
    quorum {
      
    providercorosync_votequorum
    }

    totem {
      
    cluster_nameTest
      config_version
    5
      
    interface {
        
    bindnetaddr10.10.20.60
        ringnumber
    0
      
    }
      interface {
        
    bindnetaddr10.10.19.60
        ringnumber
    1
      
    }
      
    rrp_modepassive
      ip_version
    ipv4
      secauth
    on
      version
    2
      rrp_mode
    passive
    }

     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    RokaKen likes this.
  6. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    thank you for the configuration information.

    when a node is added to the cluster -
    which address is use in ' pvecm add ' ?
     
  7. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,070
    Likes Received:
    250
    You would add a new node like this
    Code:
    pvecm add <cluster_member_ip>  --ring0_addr <local ip ring0> --ring1_addr <local ip ring1>
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    would cluster_member_ip be on vm network or ring?
     
  9. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,070
    Likes Received:
    250
    It doesn't matter in the current version.
    In the past, you should choose the ring0 address.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. RobFantini

    RobFantini Active Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,474
    Likes Received:
    18
    It will be a couple of weeks before we do the change over to dual ring network.

    I plan on trying to convert our existing corosync.conf file to use the new ring addresses . The other option would be to rebuild the production cluster . we have 7 nodes. so i could move vm's , ceph osd's and mons to 4 of them. then try converting 3 nodes to use dual ring. our cpu and memory usage is very low so 3-4 nodes could easily run everything.

    is that a path that you'd attempt to go?
     
  11. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,070
    Likes Received:
    250
    I would change the corosync.conf, because it is faster but it depends on your setup but normally it can be done online.
    The only thing is turning off HA if you use HA, before you change the corosync network.
    And be sure that the watchdog is not activated.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice