Is this proxmox network configuration bad?

[LunarMagic]

Hello,

I have this network configuration and i keep having Quorate issues randomly. I have each LACP connection on separate switches like it shows in the comments. Is it better to just not have backup network connections to keep things simple?

 

[shanreich]

Are you running corosync across those nested bonds? We strongly recommend using an unbonded, separate interface for corosync, since it is known to have issues with bonded interfaces. Also make sure you have redundant corosync networks, so you always have a fallback network if the primary one fails. The failover corosync provides is usually much faster than what a bond can provide.

Generally I'd recommend against nesting bonds like that. LACP already provides redundancy for you in case of a failure. If you really want to bond the 4 ports together why not bond all 4 of them via LACP? Do they have different bandwidth? If you want to hedge against a NIC with multiple ports failing, you can always bond ports of different NICs together via LACP.

This setup is needlessly complex imo, and could lead to some weird issues in the future.

 

[LunarMagic]

Are you running corosync across those nested bonds? We strongly recommend using an unbonded, separate interface for corosync, since it is known to have issues with bonded interfaces. Also make sure you have redundant corosync networks, so you always have a fallback network if the primary one fails. The failover corosync provides is usually much faster than what a bond can provide.

Generally I'd recommend against nesting bonds like that. LACP already provides redundancy for you in case of a failure. If you really want to bond the 4 ports together why not bond all 4 of them via LACP? Do they have different bandwidth? If you want to hedge against a NIC with multiple ports failing, you can always bond ports of different NICs together via LACP.

This setup is needlessly complex imo, and could lead to some weird issues in the future.

I believe its running on bond0 and bond1. I haven't touched it in a while but how would you go about changing what corosync uses? I didn't realize a bonded connection for corosync caused issues so thank you for that!!

So i didn't do 4 via LACP because my unifi device can only bind 2 ports together. If i do an LACP connection in Proxmox can i do it across 2 switches if thats the case or can LACP connections only be on 1 switch?

My whole goal was to have redundant connections for proxmox so if switches went down i was good and i heard LACP connections really helped improve speed

 

[shanreich]

I believe its running on bond0 and bond1. I haven't touched it in a while but how would you go about changing what corosync uses? I didn't realize a bonded connection for corosync caused issues so thank you for that!!

Usually you add a additional link to corosync which represents the new network configuration, then you configure the new link on all nodes and verify that it is working. Only after verifying this, you remove the old link and corosync should switch over to the other link you configured.

LACP is for bonding multiple ports on the same switch. If you want to bond across multiple switches you will need to use MLAG in addition.

 

[LunarMagic]

Usually you add a additional link to corosync which represents the new network configuration, then you configure the new link on all nodes and verify that it is working. Only after verifying this, you remove the old link and corosync should switch over to the other link you configured.

LACP is for bonding multiple ports on the same switch. If you want to bond across multiple switches you will need to use MLAG in addition.

So would you advise a single 10 gig connection for corosync and then a backup 10 gig connection on a separate switch? Unifi doesn't have MLAG so i guess ill stick to LACP connections but what would i even use LACP connections for if corosync doesn't like bonded connections?

I feel like i have all of these network ports and switches for nothing if i can't make the speed even faster