Redundant 'remote' bridging of bridges with tinc-vpn

[jaxx]

Hi everyone !

I'm trying to achieve something as the title might give away ;-)

I'm trying to join two logically distant VMBRs using multiple tinc connections.

To give some context, we (my client actually) have a bunch of servers in a hosting company (Online/Dedibox) using one account, clustered over a logical private network (vlan tagged 10G interfaces). We have a second account with a handful of other servers (awaiting to join the cluster), also connected through private network interfaces... The hosting company doesn't offer connecting two private networks from different accounts anymore *emoji very sad face*

So, I've tossed in joining the two over tinc links (in tap mode adding the links in the vmbr)

While I've done this quite a number of times using single links to connect 'distant' VMBRs, or star-like links to connect isolated VMBRs, I'd like to have a mesh network connecting two VMBRs that have quite a number of machines... (no more than a 2 to 2 is needed though)

Here's a simplified schematic using my best skills in ascii art ;-) (10G connections are done over a vlan on the naked interface, but I don't need to show them for simplicity)

-----------------------------------
      |     accountA    |          \ other pve servers
 vmbrX|    clustered    |vmbrX
+-----------+     +------------+
|SRV  ^^    |     |    ^^   SRV|
| A   | \   |     |   / |    B |
|     |  \  |     |  /  |      |
|     v   v |     | v   v      |
+-----------+     +------------+
      |    -\     /-    |   
      |      -\ /-tinc  |   
  tinc|       /-\       |tinc
      | tinc/-   -\     |   
      |    -       -    |   
+-----------+     +------------+
|     ^   ^ |     | ^   ^      |
|     |  /  |     |  \  |      |
|SRV  | /   |     |   \ |   SRV|
| C   vv    |     |    vv    D |
+-----------+     +------------+
 vmbrX|       new       |vmbrX
      |    accountB     |   
--------------------------------

Tinc daemons are configured directly on the host
If I have only one link up, all goes well, all hosts can each ping one another
tinc-up is a classic set interface up && brctl addif, tinc-down does a brctl delif && interface down
But if I put them all up, I seems to hit switching issues, and I even lose communication between CTs until I cut the tinc daemons

...
[Mon Oct 19 17:19:34 2020] vmbr2011: port 7(L2Mclient) entered blocking state
[Mon Oct 19 17:19:34 2020] vmbr2011: port 7(L2Mclient) entered disabled state
[Mon Oct 19 17:19:34 2020] device L2Mhuma entered promiscuous mode
[Mon Oct 19 17:19:34 2020] vmbr2011: port 7(L2Mclient) entered blocking state
[Mon Oct 19 17:19:34 2020] vmbr2011: port 7(L2Mclient) entered forwarding state
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:34 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
[Mon Oct 19 17:19:41 2020] net_ratelimit: 7141 callbacks suppressed
[Mon Oct 19 17:19:41 2020] vmbr2011: received packet on ens4f0.2011 with own address as source address (addr:2a:d4:7f:00:8e:05, vlan:0)
...

I was wondering if enabling STP could have helped, and I actually thought it was enabled by default.

Should it be a setting I should explore ? Any risk doing so ? Has anyone done anything similar
(Hosts on the clustered and ready side hold non-HA VMs/CTs (redundancy is done at a software level), but they remain critical)

Thanks in advance if anyone has tried any such setup !

JaXX./.

 

[jlebherz]

Hey,

can you post your Network Config and tinc Config?

 

[jaxx]

Hi,

Sure, at least the tasty parts:

Here's one of the hosts, they are all alike besides the static IPs and ConnectTo that vary from one host to another, and hit on public IPs
(some info hidden in respect to my client, even the private /24 ;-) )

# 10G interface to the "RPNv2" provided by the hosting company,
# and linked to the other hosts in the account
auto ens4f0
iface ens4f0 inet manual

auto vmbr2011
iface vmbr2011 inet static
  address 10.33.xx.67/24
  bridge_ports ens4f0.2011
  bridge_stp off
  bridge_fd 0


root@A:~# cat /etc/tinc/L2Mclient/tinc.conf
Name = A
Device = /dev/net/tun
AddressFamily = ipv4
Mode = switch
ConnectTo = C
ConnectTo = D
root@A:~#

Nothing too fancy, bridge_stp is off (as documented everywhere), but no clue how turning it on would affect the CTs and VMs and if this would help anyhow.


Thanks