Multicast - Corosync with RRP

Discussion in 'Proxmox VE: Networking and Firewall' started by TwiX, Sep 11, 2018.

  1. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    Hi,

    Let's admit we have a 3 nodes cluster.
    Each node have 2 interfaces, first one (vmbr0) for VMs + corosync and the second one for corosync (via RRP in passive mode)
    Node 1
    Interface 1 : 192.168.0.10/24 GW 192.168.0.1
    Interface 2 : 10.0.0.1/24 no GW

    Node 2
    Interface 1 : 192.168.0.11/24 GW 192.168.0.1
    Interface 2 : 10.0.0.2/24 no GW

    Node 3
    Interface 1 : 192.168.0.12/24 GW 192.168.0.1
    Interface 2 : 10.0.0.3/24 no GW

    In order to use RRP with corosync I added a new ring related to interface 2 in corosync.conf

    I made a lot of tests using omping (omping -c 10000 -i 0.001 -F -q 10.0.0.1 10.0.0.2 10.0.0.3).

    I saw a huge traffic (~10 Mbits) on interface 1 and not on interface 2, I guess because of the 239.0.0.0/8 (trying to reach the GW ?

    Is it normal ?

    Thanks a lot !
     
  2. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    202
    Likes Received:
    12
    The traffic goes out of interface 1, because this is your primary ring
     
  3. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    But I launch omping only with IPs of interface 2
     
  4. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    202
    Likes Received:
    12
    You're right - the omping traffic (unicast and multicast) should go via interface 2.
    omping uses udp-packets to 232.43.211.234:4321 by default to check multicast access.

    maybe you had another omping instance running on interface 1 ?
    you could try to use another multicast address from 232.0.0.0/8 (-m switch to omping - see man omping)

    else the output of `ip route show`, `ip addr show` , `ip maddr show` (and `ip link show`) should give you some insight
     
  5. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    No only one instance of omping.
    Ok so nodes tried to reach 232.43.211.234 by the way must use the default GW which is directly routed on interface 1 ?
     
  6. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    Here is my conf and routes.

    vmbr0 on bond0 (LACP layer 3+4) - 10.192.5.32/27
    bond1 (LACP layer 3+4) - 10.198.0.0/24

    root@dc-prox-09:~# route
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    default 10.192.5.33 0.0.0.0 UG 0 0 0 vmbr0
    10.192.5.32 0.0.0.0 255.255.255.224 U 0 0 0 vmbr0
    10.198.0.0 0.0.0.0 255.255.255.0 U 0 0 0 bond1

    root@dc-prox-09:~# omping -c 10000 -i 0.001 -F -q 10.198.0.34 10.198.0.35 10.198.0.36
    10.198.0.35 : waiting for response msg
    10.198.0.36 : waiting for response msg
    10.198.0.36 : joined (S,G) = (*, 232.43.211.234), pinging
    10.198.0.35 : joined (S,G) = (*, 232.43.211.234), pinging
    10.198.0.35 : given amount of query messages was sent
    10.198.0.36 : given amount of query messages was sent

    10.198.0.35 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.029/0.066/0.857/0.026
    10.198.0.35 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.032/0.069/0.834/0.028
    10.198.0.36 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.030/0.070/0.202/0.026
    10.198.0.36 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.033/0.078/0.566/0.028

    Seems that traffic also goes up on bond 1. but clearly goes up on bond0
    [​IMG]
     
  7. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    Just took a trace of bond0 with tcpdump when running omping.
    Graph indicates that there's a lot of incoming traffic but in pcap I found lots of outgoing packets from 10.198.0.0/24 to multicast address

    [​IMG]
     
  8. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    Also what I noticed, mac_address used to send traffic to 232.43.211.234 correspond to bond1 but was send by bond0 ?

    Code:
    ip link show
    10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
        link/ether d0:94:66:5a:3f:1a brd ff:ff:ff:ff:ff:ff
    
    12: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
        link/ether f4:e9:d4:d6:dc:d0 brd ff:ff:ff:ff:ff:ff
    [​IMG]
     
  9. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    202
    Likes Received:
    12
    please try to use a different mcast address (e.g. 232.0.0.1) for omping (-m switch).
    please post the output of:
    * `ip addr show`
    * `ip maddr show`

    * do you use the same switch for both lacp links? (if it is not multicast aware, and has no listener configured it would maybe send out all multicast packets on all interfaces)
     
  10. TwiX

    TwiX Member
    Proxmox VE Subscriber

    Joined:
    Feb 3, 2015
    Messages:
    111
    Likes Received:
    1
    Thanks, that makes sense. I will investigate on my netgear switch stack
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice