Problem with corosync and/or multicast?

[j4ys0n]

Problem: When I connect multiple nodes to my MikroTik CRS305-1G-4S+IN switch, the nodes stop communicating reliably. The MikroTik switch can actually be a router or switch - there are two embedded OSes (RouterOS and SwitchOS) I've tried both, but am currently using SwitchOS.


I have 4 Proxmox 6 nodes. Right now they're all connected to a ubiquiti network managed switch and everything works as expected. I'm working on adding some 10G ethernet equipment to get more throughput between at least 2 of my nodes. So I bought a MikroTik 10G switch and a couple 10G NICs. The MikroTik switch is plugged into the main switch. But when I plug the nodes into the MikroTik switch, they stop communicating with each other and the other nodes. This doesn't happen when the same nodes are plugged into a regular 1g unmanaged switch (basically, if i just swap out the MikroTik switch for a regular switch). I'm not sure what's going wrong. I do have IGMP snooping enabled on the ubiquiti network (not sure if that matters). It seems like there's some sort of network conflict between the ubiquiti equipment and the MikroTik equipment.

Any insight is greatly appreciated.

 

[wolfgang]

Hi,

I would say the problem is the MikroTik switch.
Proxmox VE 6 and corosync 3 use UDP unicast and not multicast.

 

[j4ys0n]

got it. yeah i would agree that the problem is the MikroTik switch - any idea what I should look into to fix it?

edit:
here's the switch documentation. https://wiki.mikrotik.com/wiki/SwOS/CRS3xx#System

i turned on IGMP snooping, no idea if that will help though. will try again this evening.

 

[wolfgang]

Ask the Mikrotik guys.

 

[j4ys0n]

yes the issue is the switch, but it's with proxmox communication - i'm not even sure what to ask. if the nodes don't reliably communicate, is it a problem with corosync?

 

[guletz]

Ask the Mikrotik guys.

... and a Mikrotik guy respond ... as a Mikrotik certified guy(mtcre, mtcse and more then 100 mikrotik devices in my backyard) I can say that in a Mikrotik only network (routeros only) I have a 2 nodes using pmx 6.x without any problem starting from last year (2 nodes cluster).

The same experience with Proxmox 5.x and Mikrotik only network (routeros only). For me this combination is a win win case

And I use also use ospf who use multicast as Proxmox. I can only speculate ... maybe is some firewall problem somewhere ?

Why you do not try to put all your Pmx nodes in the Mikrotik switch and see if your 4 nodes ar ok? Then go forward conecting the mikrotk switch to the LAN ?





Good luck / Bafta.

 

[guletz]

yeah i would agree that the problem is the MikroTik switch

Wrong .... maybe you have done somthing wrong(RouterOS case) ! And as a side note, I see some others pmx users who use mikrotik without any multicast/corosync problems ...and I read a lot of this nice Forum threads daily

Good luck / Bafta !

 

[j4ys0n]

... as a Mikrotik certified guy(mtcre, mtcse and more then 100 mikrotik devices in my backyard) I can say that in a Mikrotik only network (routeros only) I have a 2 nodes using pmx 6.x without any problem starting from last year (2 nodes cluster).

The same experience with Proxmox 5.x and Mikrotik only network (routeros only). For me this combination is a win win case

And I use ospf who use multicast as Proxmox. I can only speculate ... maybe is some firewall problem somewhere ?




Good luck / Bafta.

this is good to know! thanks for the response!

any idea what settings i should check for?

i'm using the 4 port 10gbit switch/router with sfp+ to rj45 connectors. if i plug 1 node into the device, everything is ok. if i plug 2 nodes into the device, they stop communicating with each other and the rest of the network.

i'm fine switching over to routeros if that needs to be done - i just thought switchos would be better for this situation since i already have a router/network controller and this is basically acting as a higher throughput switch.

 

[guletz]

any idea what settings i should check for?

Start use routerOS. In this case I can help you. I know some triks on RouterOS and Proxmox. Also I have a new project with Proxmox and your Mikrotik switch that I do not have it yet. Use the conversention from Forum to send me your mail and I will try to help.

 

[j4ys0n]

Might be resolved now. Seems to be working fine anyway - I moved the switch and enabled IGMP snooping, that might have done it as that's enabled on the rest of the network. Everything else is pretty much the same.

 

[voklav]

I failed to make my first cluster and that's it :/
I struggled with this problem for two weeks.
I was almost giving up.
Thx to you j4ys0n everything works now!

enable IGMP snooping in bridge interface on Mikrotik ^^ and nodes to start to communicate like old friends immediately! Fantastic!

 

[spirit]

I failed to make my first cluster and that's it :/
I struggled with this problem for two weeks.
I was almost giving up.
Thx to you j4ys0n everything works now!

enable IGMP snooping in bridge interface on Mikrotik ^^ and nodes to start to communicate like old friends immediately! Fantastic!

old proxmox version ? (<= 5 ?). Multicast is not used anymore since corosync 3 (proxmox >= 6)

 

[voklav]

old proxmox version ? (<= 5 ?). Multicast is not used anymore since corosync 3 (proxmox >= 6)

Nope.
Two hardware are fresh installed: VE 7.1-12