Multicast - Corosync with RRP

[TwiX]

Hi,

Let's admit we have a 3 nodes cluster.
Each node have 2 interfaces, first one (vmbr0) for VMs + corosync and the second one for corosync (via RRP in passive mode)
Node 1
Interface 1 : 192.168.0.10/24 GW 192.168.0.1
Interface 2 : 10.0.0.1/24 no GW

Node 2
Interface 1 : 192.168.0.11/24 GW 192.168.0.1
Interface 2 : 10.0.0.2/24 no GW

Node 3
Interface 1 : 192.168.0.12/24 GW 192.168.0.1
Interface 2 : 10.0.0.3/24 no GW

In order to use RRP with corosync I added a new ring related to interface 2 in corosync.conf

I made a lot of tests using omping (omping -c 10000 -i 0.001 -F -q 10.0.0.1 10.0.0.2 10.0.0.3).

I saw a huge traffic (~10 Mbits) on interface 1 and not on interface 2, I guess because of the 239.0.0.0/8 (trying to reach the GW ?

Is it normal ?

Thanks a lot !

 

[Stoiko Ivanov]

The traffic goes out of interface 1, because this is your primary ring

 

[TwiX]

But I launch omping only with IPs of interface 2

 

[Stoiko Ivanov]

You're right - the omping traffic (unicast and multicast) should go via interface 2.
omping uses udp-packets to 232.43.211.234:4321 by default to check multicast access.

maybe you had another omping instance running on interface 1 ?
you could try to use another multicast address from 232.0.0.0/8 (-m switch to omping - see man omping)

else the output of `ip route show`, `ip addr show` , `ip maddr show` (and `ip link show`) should give you some insight

 

[TwiX]

No only one instance of omping.
Ok so nodes tried to reach 232.43.211.234 by the way must use the default GW which is directly routed on interface 1 ?

 

[TwiX]

Here is my conf and routes.

vmbr0 on bond0 (LACP layer 3+4) - 10.192.5.32/27
bond1 (LACP layer 3+4) - 10.198.0.0/24

root@dc-prox-09:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.192.5.33 0.0.0.0 UG 0 0 0 vmbr0
10.192.5.32 0.0.0.0 255.255.255.224 U 0 0 0 vmbr0
10.198.0.0 0.0.0.0 255.255.255.0 U 0 0 0 bond1

root@dc-prox-09:~# omping -c 10000 -i 0.001 -F -q 10.198.0.34 10.198.0.35 10.198.0.36
10.198.0.35 : waiting for response msg
10.198.0.36 : waiting for response msg
10.198.0.36 : joined (S,G) = (*, 232.43.211.234), pinging
10.198.0.35 : joined (S,G) = (*, 232.43.211.234), pinging
10.198.0.35 : given amount of query messages was sent
10.198.0.36 : given amount of query messages was sent

10.198.0.35 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.029/0.066/0.857/0.026
10.198.0.35 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.032/0.069/0.834/0.028
10.198.0.36 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.030/0.070/0.202/0.026
10.198.0.36 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.033/0.078/0.566/0.028

Seems that traffic also goes up on bond 1. but clearly goes up on bond0

 

[TwiX]

Just took a trace of bond0 with tcpdump when running omping.
Graph indicates that there's a lot of incoming traffic but in pcap I found lots of outgoing packets from 10.198.0.0/24 to multicast address

 

[TwiX]

Also what I noticed, mac_address used to send traffic to 232.43.211.234 correspond to bond1 but was send by bond0 ?

ip link show
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether d0:94:66:5a:3f:1a brd ff:ff:ff:ff:ff:ff

12: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f4:e9:d4:d6:dc:d0 brd ff:ff:ff:ff:ff:ff

 

[Stoiko Ivanov]

please try to use a different mcast address (e.g. 232.0.0.1) for omping (-m switch).
please post the output of:
* `ip addr show`
* `ip maddr show`

* do you use the same switch for both lacp links? (if it is not multicast aware, and has no listener configured it would maybe send out all multicast packets on all interfaces)

 

[TwiX]

Thanks, that makes sense. I will investigate on my netgear switch stack