SDN overlay network in routed mesh setup

[fmaurer]

Hello,
I have a three-node cluster with two rings.
1. One full ring between the three nodes. Similar to the configuration shown here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Example


2. And one "uplink" ring of the "e0np0" interfaces as shown here (derived picture from the wiki-topology with the additional ring):

      (WAN) |       ┌───────────┐        | (WAN)
            |       │   Node2   │        |
            |       ├─────┬─────┤        |
            |     ┌─┤e0np0│e1np1├─┐      |
            |     | ├─────┬─────┤ |      |
            |     | │eno21│eno10│ |      |
            |     | └──┬──┴──┬──┘ |      |
            |     │    │     │    |      |
┌───────┬─────┐   │    │     │    │   ┌─────┬───────┐
│       │e0np0│   │    │     │    │   │e0np0│       │
│       ├─────┤   │    │     │    │   ├─────┤       │
│       │e1np1├───┘    │     │    └───┤e1np1|       │
│       ├─────┤        │     │        ├─────┤       │
│       │eno10├────────┘     └────────┤eno21│       │
│ Node1 ├─────┤                       ├─────┤ Node3 │
│       │eno21├───────────────────────┤eno10│       │
└───────┴─────┘                       └─────┴───────┘

Image with the full topology (forget about the IPMI network here)



So as ceph is working well with this setup, just as documented here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Example

The question which is still open is:

How to manage the VM IP Addresses running on these hosts.
1. node1 and node3 have direct access to the uplink - can therefore put VMs into their vmbr0 and be done
2. node2 does not have direct access and must be routed via epyc03 and/or epyc01 - the VMs on this host should be able to use public IPs from the uplink broadcast domain.

How we currently solve this:
1. node03 puts e0np0 and e1np1 into vmbr0
2. node02 puts e1np1 into vmbr0
3. all VMs are on the respective vmbr0

Alternative option:
- just buy a switch and get proper uplink configured for node02 as well

I wonder what options are possible for such a setup. I prefer using onboard SDN EVPN stuff, but am also interested in other options and ideas for this problem.

 

[shanreich]

If you're willing to use an EVPN zone, then you can designate the two nodes with the uplinks as exit nodes. They will then announce a default route to the node02 and all traffic from node02 will be routed towards them.

 

[fmaurer]

Hi @shanreich, this would require that I communicate BGP with the upstream router and completely change the setup in that direction, right?
node1 and node3 are currently in the same L2 as the upstream router (which directly routes the public IPv4 /27 network and allows the VMs to access it).

I wonder if there is an elegant solution which still allows to have the public IPs in the VMs (so no SNAT), but does not require a change to the router on the two WAN ports. I guess that's just not possible.
Some other posts mentioned ProxyARP - so that the exit nodes impersonate the IP of the VMs of node02 - but this would not solve the redundancy and does not look like a solid solution (https://wiki.debian.org/BridgeNetworkConnectionsProxyArp).