New Servers w/100G Trunks, Should I still use a separate Corosync network?

[helojunkie]

So, as the title says, I am deploying all new Proxmox servers to replace our aging fleet of 2U Dells. Currently, I have a 10G trunk for all of my normal VLANs and a separate 10G connection specific to only Corosync VLAN traffic. My new servers have 4 x 10G NICs and 2 x 100G NICs each.

I was thinking of just using 1 x 100G link and include my Corosync VLAN in that trunk, and I cannot seem to find anything that would suggest that would not work just fine, but I thought I would reach out and see if anyone has any real-world thoughts on this type of deployment or suggestions.

Thank You

 

[CH.illig]

in my mind...

10G or 100G don't care ;-)

but i still would use a separate network for stability

i would use

2x100G in active backup / or any else HA setting your switch is supporting
-> VLAN for all VM's
-> VLAN for Corosync second network

2x 10G in active backup / or any else HA setting your switch is supporting
-> VLAN for your Host access
-> VLAN for Corosync first Network

anything change with ceph in use ... but that's how i would use this network cards...

But yes, you don't have to ... its just if something happened on one NIC
a dedicated NIC for Corosync that keep your cluster ready is nice...

 

[sb-jw]

I've only had 2x 10 GbE per node for years and no problems with Corosync. But I wouldn't bring a server online with anything other than 802.3ad. In my opinion, two switches in the MLAG are mandatory.

The recommendation is very clear that everything should be separated, but then we are also talking about external, internal, cluster, management, storage and live migration. But do you want 12 cables per node?

Ultimately, everyone will probably have their own experiences and say something different, I feel comfortable with 2x 10 GbE and wouldn't give it up so quickly.

 

[spirit]

you just need to be sure to not saturate links.

If you saturate links, (even on only 1node), it can break the whole cluster.

I have tried recently, doing a ddos with flooding 40gbit/s on a 40gits link in inbound only, and the whole cluster and /etc/pve was locked.

 

[helojunkie]

Thanks to everyone for their feedback. For years I have run a 6-node cluster on the two 10G connections, one trunked for all vlans, the other dedicated for Corosync and have never had an issue.

In reading the replies, I think I will stick with a dedicated 10G corosync connection!

Thanks Everyone for the feedback!