PM 6 multicast related bugs

[mailinglists]

HI,

it took me some time, but I finally understood that is going on.
Before I open bugs on bugzilla, I want to make sure I get things right.

1. BUG: Proxmox docs state, that PM 6 cluster does not require multicast.

"Note: Proxmox VE 6.0 uses corosync 3 which switched out the underlying transport stack with Kronosnet (knet). Kronosnet currently only supports unicast."
https://pve.proxmox.com/wiki/Multicast_notes

However during a joining of a new node, it actually sends multicast packets! So docs should clearly state that!
Which brings us to bug numero 2.

2. Proxmox bridge blocks multicast by default.

As Proxmox uses multicast packets for joining node into the cluster, it should by default allow them.
If one set's Proxmox cluster network on a Linux bridge, it will not join cluster, because Linux bridge will block multicast traffic.
If the same node (after rollback) is added again, but this time using a physical interface, it joins perfectly.

Yeah and I know what igmp snooping is and it is disabled on my switches for cluster communication.
It guess it has even been disabled by default in linux bridge in first PM 6.* releases, however somewhere along the line, you changed that default behaviour and it blocks now.

(0.53) #show igmpsnooping
Admin Mode..................................... Disable
Multicast Control Frame Count.................. 0
IGMP Router-Alert check........................ Disabled
Interfaces Enabled for IGMP Snooping........... None
VLANs enabled for IGMP snooping................ None

root@p35:~# cat /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping
1
root@p35:~# cat /sys/devices/virtual/net/vmbr0/bridge/multicast_querier
0

So I would suggest that for 1. fix the docs and for 2. enable querier by default or disable snooping by default or fix the docs and say, PM clusters are not supported when cluster ring is on a bridge.

Should I open bug reports now?

 

[mailinglists]

@Stoiko Ivanov @dcsapak tags.

 

[Pierre-Yves]

If one set's Proxmox cluster network on a Linux bridge, it will not join cluster, because Linux bridge will block multicast traffic.

That is to say that you must have a dedicated network interface for cluster network ?
A classical way to build Promox, in my opinion, is to connect SAN on a side, and on the other side 2 high speed nics (10 Gbs mini), each one of them connected to a high speed switch that runs with backup or balancing mode.
So you've got :
SW1-----ETH1------ \ I I----BOND ----- BRIDGE --- VLANS SW2-----ETH2-------/

So this is not running for cluster any longuer ?

 

[spirit]

However during a joining of a new node, it actually sends multicast packets! So docs should clearly state that!
Which brings us to bug numero 2.

do you have any tcpdump/wireshark trace ? that's really strange....

2. Proxmox bridge blocks multicast by default.

only if you don't have an igmp querier on your network.




That's really strange, I don't have any igmp querier anymore on my network, and keep default bridge igmp values, and I don't have any node join problem. (I have really don't remember to have see multicast corosync traffic on node join)

 

[dcsapak]

do you have any tcpdump/wireshark trace ? that's really strange....

yes a package capture would be interesting

 

[mailinglists]

Hi guys, sorry for delay, but "life" happens.

@Pierre-Yves i think this is still the way to go. I just create bridge after joining cluster.

@dcsapak @spirit
Now I'm not sure, if it still it has something to do with with bridge multicast traffic, but it certainly is bridge related.
So I will do what you ask. Will add another node shorty, when it's IP is on a bridge instead of directly on interface and do tcpdump on both sides.
If it fails, I will remove the interface from the bridge and speficy IP directly on it, and join the node with tcpdump running as well.

Now we have added two servers to the cluster, buy configuring bridge later and they work normally since joining.

 

[mailinglists]

A little update. I added another node yesterday, logging tcpdump on bridges and physical cards, but it joined without a problem.
Will log the same when adding another node, in case if the error pops up again. I will have and share the tcpdumps.