host loses cluster when moving IP to another interface/vmbr

[Seabob]

Dear all,
I got a Dell R730 with 4x 1GbE onboard (interfaces eno1 to eno4) and two additional 10GbE NICs (interfaces enp129 and enp133). I installed my host using eno1 and bound IP-Address 192.168.2.35 onto it, or better: this address was bound to vmbr0, which uses eno1. I used this connection to register myhost to the cluster.

Moving forwarding I created bond0 (balance-alb) from enp129 and enp133. For testing I also created vmbr1 on top of bond0 and put address 192.168.2.36 onto it.
I can ping across the entire company and the internet and both addresses respond, when I ping them. So, there should not be an underlying issues with our network or with the created bond0.

For reasons of fault-tolerance and performance I like to run the host-IP 192.168.2.35 on the bond rather than on a single NIC, so I changed IP-addresses of vmbr0 and vmbr1, which works. For my understanding I should be fine disconnecting eno1/vmbr0 from the network now, but whenever I do so, the host loses connection to the cluster, while its VMs continue to run fine on vmbr1.
In a further step I even assigned bond0 to vmbr0 and IP 192.168.2.35. Eno1 is now connected to vmbr1 with IP 192.168.2.36, but whenever I disconnect eno1, the host loses connection to the cluster.

What do I overlook or could have done better? - Goal is still to run the host using a fault-tolerant bond for LAN-access and cluster communication.

 

[sb-jw]

If your goal is performance and availability, then you should also have a few euros for switches with MLAG and LACP. You can get them today for 500 EUR.
Personally, I would stay away from anything that is not 802.3ad. Not everything that works is good, so it's better to use a standard that is designed for this.

But I suspect that your problem lies in the balance-alb. As far as I remember, balance-alb distributes the traffic in both directions on the interfaces. So you should go to balance-tlb as a test. However, you should check whether you can use 802.3ad, then you won't have such problems.

 

[Seabob]

If your goal is performance and availability, then you should also have a few euros for switches with MLAG and LACP. You can get them today for 500 EUR.
Personally, I would stay away from anything that is not 802.3ad. Not everything that works is good, so it's better to use a standard that is designed for this.

But I suspect that your problem lies in the balance-alb. As far as I remember, balance-alb distributes the traffic in both directions on the interfaces. So you should go to balance-tlb as a test. However, you should check whether you can use 802.3ad, then you won't have such problems.

@sb-jw: The 1000€-switches I use are not stackable, so I cannot span an LACP-trunk across 2 switches. The strange behaviour also persists when using "active/backup" bond.
Anyway, there are no issues or dropped packets with the VMs running on the bond, appearently the host-communication is somehow fixed on eno1, this is my issue.

btw: Connecting 2 wires to one switch just moves the single point of failure to the switch, but I like to push it out a bit farther (even on cost of performance).

 

[sb-jw]

Are both switches connected to each other? Or have you made sure that the active port of all servers is on the same switch. Otherwise the servers won't see each other. To make sure that doesn't matter, you can solve this with an interconnect between both switches, but be careful not to create a loop.

 

[Seabob]

Are both switches connected to each other? Or have you made sure that the active port of all servers is on the same switch. Otherwise the servers won't see each other. To make sure that doesn't matter, you can solve this with an interconnect between both switches, but be careful not to create a loop.

Both switches "see" each other by their uplink to the backbone, so I avoid loops and I can see the VMs don't have the slightest trouble with it, which needed to be the case, if I had a bonding/switching issue.

But again, this also occurs when using "active-backup", which doesn't require any precautions on switches and means there is only one active MAC-address. And no, I'm looking for a setup which avoids connecting all ports to just one switch.

One other host in the cluster is connected to the same switch, where the bond and eno1 are connected to. The third host is connected to another switch on the backbone. Under all circumstances host 2 and 3 see each other, while host 1 shows the issue.
If I recall things right, then I had this issue also, when trying to use "eno2" (no bonding) instead of eno1.

I triple-checked VLAN-tags, PVIDs and related settings on the switches, so we can skip this area, too.

 

[sb-jw]

Both switches "see" each other by their uplink to the backbone, so I avoid loops and I can see the VMs don't have the slightest trouble with it, which needed to be the case, if I had a bonding/switching issue.

I would never consider an uplink from the data center as a connection within my infrastructure. To do this, you set up cross connects that run independently of the data center or you have your own core network, which also only receives the uplink from the data center and you do everything internally there yourself. This can potentially be a problem for your setup.

But again, this also occurs when using "active-backup", which doesn't require any precautions on switches and means there is only one active MAC-address. And no, I'm looking for a setup which avoids connecting all ports to just one switch.

With active-backup, the active ports of all servers should be on the same switch. All inactive ports correspondingly on the other. If you cannot ensure this, you should create an interconnect between the two switches. The cluster VLAN should then run on this interconnect, so that you can enable communication from both switches with all servers, regardless of which one the active port is on. But this also means that the interconnect becomes your bottleneck.
If you want to implement exactly this concept via the uplinks, you also have to make sure that the data and your VLAN are actually on the data center infrastructure. However, I would definitely advise you not to put the internal network with the external traffic in the same VLAN.

The third host is connected to another switch on the backbone.

So you have the servers distributed across the different racks and all traffic goes through the uplink ports? I definitely can't recommend that either. If there are 40 other customers on a switch and one of them gets a DDoS, which then maxes out the switch uplink, then your server has a problem and your cluster loses connections. Even a few ms of additional latency can cause the Corosync connection to no longer work properly.
You should really rethink your concept and make sure that all nodes of a cluster are in the same rack and that an MLAG switch with 802.3ad is installed there. This will save you some problems and headaches now and will save you from bigger problems in the future.

Under all circumstances host 2 and 3 see each other, while host 1 shows the issue.
If I recall things right, then I had this issue also, when trying to use "eno2" (no bonding) instead of eno1.

In my opinion, you shouldn't recall it from your memory but rather evaluate it and document the results. You should really make sure that all configurations are identical, that the ports have the correct VLAN, are not deactivated, that they are also allowed to talk to the uplink, etc. pp.

 

[Seabob]

I would never consider an uplink from the data center as a connection within my infrastructure. To do this, you set up cross connects that run independently of the data center or you have your own core network, which also only receives the uplink from the data center and you do everything internally there yourself. This can potentially be a problem for your setup.

With active-backup, the active ports of all servers should be on the same switch. All inactive ports correspondingly on the other. If you cannot ensure this, you should create an interconnect between the two switches. The cluster VLAN should then run on this interconnect, so that you can enable communication from both switches with all servers, regardless of which one the active port is on. But this also means that the interconnect becomes your bottleneck.
If you want to implement exactly this concept via the uplinks, you also have to make sure that the data and your VLAN are actually on the data center infrastructure. However, I would definitely advise you not to put the internal network with the external traffic in the same VLAN.

So you have the servers distributed across the different racks and all traffic goes through the uplink ports? I definitely can't recommend that either. If there are 40 other customers on a switch and one of them gets a DDoS, which then maxes out the switch uplink, then your server has a problem and your cluster loses connections. Even a few ms of additional latency can cause the Corosync connection to no longer work properly.
You should really rethink your concept and make sure that all nodes of a cluster are in the same rack and that an MLAG switch with 802.3ad is installed there. This will save you some problems and headaches now and will save you from bigger problems in the future.

In my opinion, you shouldn't recall it from your memory but rather evaluate it and document the results. You should really make sure that all configurations are identical, that the ports have the correct VLAN, are not deactivated, that they are also allowed to talk to the uplink, etc. pp.

Sorry, but this is completely leading away from my question. The present infrastructure is like it is and my management won't invest for the moment, so I have to get best out of it. But thanks for your hints, I'll keep them in mind.

As I mentioned above:
The issue is already present when trying to change vmbr1 from single NIC eno1 to eno2, connected to the same switch, or when trying to change vmbr0 from bond0 to single enp129. Where is the switching/bonding issue here?

So please let us return to my inital question: why is the cluster communication stuck on "eno1", even if I put the registered IP-address to some other interface? Maybe we cover additional issues later.

 

[sb-jw]

why is the cluster communication stuck on "eno1", even if I put the registered IP-address to some other interface?

Is not it. Either you made a mistake when changing the network configuration or, as mentioned several times above, you simply have a fundamental problem in your network topology.

A first option would be to post all network configurations here.

 

[Seabob]

Observation:
when joining a host to a cluster you also configure the link the host will use. This was the issue.

Lesson learned:
make sure to have any network-related (and probably storage-related) configuration finished and tested prior joining a host to a cluster.

 

[sb-jw]

when joining a host to a cluster you also configure the link the host will use. This was the issue.

Can you elaborate on this?