help needed to enable multicast

[Waschbüsch]

Hi all,

I have the following cluster setup:

3 nodes
a Netgear XS712T 10G switch

node 1 uses port 1 on the switch, node 2 port 2, etc.

port 12 on the switch is my uplink to the internet.

I use vlans to seperate different kinds of traffic.

vlan 5 is for internal PVE traffic with ports 1, 2 and 3 as members (tagged)
vlan 10 is for public traffic and ports 1-3 (tagged) as well as 12 (untagged) are the members

On each node, I use openvswitch to configure my network like this:

auto lo
iface lo inet loopback

allow-ovs vmbr0

allow-vmbr0 eth2
iface eth2 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0
        ovs_options vlan_mode=native-untagged trunks=5,10
        mtu 9000
# 10GBit Interface

allow-vmbr0 vlan10
iface vlan10 inet static
        address  xxx.xxx.xxx.xxx
        netmask  yyy.yyy.yyy.yyy
        gateway  zzz.zzz.zzz.zzz
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=10
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        mtu 1500
# VLAN 10: public traffic

iface eth0 inet manual
# 1 GBit interface, unused

iface eth1 inet manual
# 1 GBit interface, unused

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports eth2 vlan5 vlan10
        mtu 9000
# Bridge for internal VLANs 5 and 10 as well as all VM interfaces

allow-vmbr0 vlan5
iface vlan5 inet static
        address  192.168.1.xxx
        netmask  255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=5
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        mtu 9000
# VLAN 5: cluster traffic

Now, so far I have been unable to enable multicast.
The switch supports IGMP snooping AND querying, but I have been unable to get any multicast packets through.
omping will always show 100% loss for multicast.

Any ideas what the reason might be?

 

[manu]

Hi
Have you read https://pve.proxmox.com/wiki/Multicast_notes ?
If it still does not work, and you are in a safe network, you can disable IGMP snooping alltogether, which removes the need to configure a IGMP querier.

 

[Waschbüsch]

Thanks for the feedback. I had consulted https://pve.proxmox.com/wiki/Multicast_notes, yes, but to no avail.
When you say I could do without IGMP snooping, would that not mean falling back to unicast? Or do I have my networking facts all mixed up?

 

[manu]

IIRC:
If you have IGMP snooping active, you need a IGMP querier to get multicast working.

If you have IGMP snooping deactivated, multicast traffic will be sent to all ports of the switch, and you don't need an IGMP querier.

 

[Waschbüsch]

But should omping not show that multicast packets got through?
It doesn't. It says 100% package loss and only unicast got through...

 

[rahul1985joshi]

I have 11 node cluster, with one of them is connected to trunk port of cisco switch igmp enabled. All nodes remain in quorum but only one node connected to trunk port of cisco switch get out of quorum after 2-5 mins. I have to restart corosync service then it again comes in cluster quorum.
Please find below details-
Corosync logs--
Dec 21 16:19:03 pve-01 corosync[1630]: error [TOTEM ] FAILED TO RECEIVE
Dec 21 16:19:03 pve-01 corosync[1630]: [TOTEM ] FAILED TO RECEIVE
Dec 21 16:19:11 pve-01 corosync[1630]: notice [TOTEM ] A new membership (10.10.10.21:23704) was formed. Members left: 2 3 4 11 10 9 5 6 7 8
Dec 21 16:19:11 pve-01 corosync[1630]: notice [TOTEM ] Failed to receive the leave message. failed: 2 3 4 11 10 9 5 6 7 8
Dec 21 16:19:11 pve-01 corosync[1630]: warning [CPG ] downlist left_list: 10 received
Dec 21 16:19:11 pve-01 corosync[1630]: notice [QUORUM] This node is within the non-primary component and will NOT provide any services.
Dec 21 16:19:11 pve-01 corosync[1630]: notice [QUORUM] Members[1]: 1
Dec 21 16:19:11 pve-01 corosync[1630]: notice [MAIN ] Completed service synchronization, ready to provide service.
Dec 21 16:19:11 pve-01 corosync[1630]: [TOTEM ] A new membership (10.10.10.21:23704) was formed. Members left: 2 3 4 11 10 9 5 6 7 8
Dec 21 16:19:11 pve-01 corosync[1630]: [TOTEM ] Failed to receive the leave message. failed: 2 3 4 11 10 9 5 6 7 8
Dec 21 16:19:11 pve-01 corosync[1630]: [CPG ] downlist left_list: 10 received
Dec 21 16:19:11 pve-01 corosync[1630]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Dec 21 16:19:11 pve-01 corosync[1630]: [QUORUM] Members[1]: 1
Dec 21 16:19:11 pve-01 corosync[1630]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 21 16:19:11 pve-01 pmxcfs[1604]: [status] notice: node lost quorum

/etc/network/interfaces

auto lo
iface lo inet loopback
auto enp2s0
iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.10.10.21
netmask 255.255.255.0
gateway 10.10.10.1
bridge-ports enp2s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

 

[rahul1985joshi]

I have resolved my issue.

echo 0 > /sys/class/net/vmbr0/bridge/multicast_snooping

Disabled multicast-snooping & Voillaaa its automatically joined cluster without restarting corosync service. it is stable in cluster now.

 

[Tozz]

echo 0 > /sys/class/net/vmbr0/bridge/multicast_snooping

Disabled multicast-snooping & Voillaaa its automatically joined cluster without restarting corosync service. it is stable in cluster now.

Thanks. This instantly resolved our issues with IPv6 neighbor discovery on VMs running on our platform!
We had intermittend issues with machines unable to reach machines within the same subnet over IPv6.

Disabling multicast_snooping on the hypervisor fixed it!