[SOLVED] Struggling with Proxmox EVPN with BGP

[Pinkbyte]

Hi. I am currently struggling with setting up Proxmox EVPN with BGP in my homelab.

Topology so far:

                                ---- ainz (first node) 192.168.6.11
                              /
Mikrotik router  192.168.6.1  ------- albedo (second node) 192.168.6.12
                              \
                                ----- demiurge (third node) 192.168.6.13

Proxmox VE version: 8.2.4 on all nodes.

The plan is to create some networks under EVPN and send routes for them to Mikrotik via BGP (so SNAT is NOT used).

Configuration of EVPN itself is in evpn.png attachment ('Primary exit node' is empty)
Configuration of virtual network(vnet100) under EVPN is in vnet.png attachment
Configuration of one of BGP controller is on bgp.png attachment (others are identical, except for node option)

For testing purposes firewall on Mikrotik is disabled (everything is allowed).

I have two VMs (test1 and test2).
- test1(192.168.100.11/24) is sitting on ainz(first node)
- test2(192.168.100.12/24) is sitting on albedo(second node)

Both VMs pings each other and default gateway of EVPN network(192.168.100.1).
So far, so good.

Trying to ping Mikrotik from test1 - fail
Trying to ping Mikrotik from test2 - fail

Routes in Mikrotik:

[admin@rb] > ip route print where bgp and dst-address in 192.168.100.0/24
Flags: D - DYNAMIC; A - ACTIVE; b - BGP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
    DST-ADDRESS        GATEWAY       DISTANCE
DAb 192.168.100.0/24   192.168.6.11        20
D b 192.168.100.0/24   192.168.6.13        20
D b 192.168.100.0/24   192.168.6.12        20
D b 192.168.100.11/32  192.168.6.13        20
DAb 192.168.100.11/32  192.168.6.12        20
DAb 192.168.100.12/32  192.168.6.11        20
D b 192.168.100.12/32  192.168.6.13        20

BTW, why route table contains routes for test1 VM to all nodes, except the node where test1 VM is located (same is for test2 VM) ?!!

If i, for example, disable BGP sessions on Mikrotik for albedo(second node) and demiurge(third node), test1 VM(running on ainz) can ping Mikrotik, test2 VM still does not.
Same thing happens if i disable BGP sessions on Mikrotik for ainz and demiurge, leaving only albedo - test2 VM(running on albedo) can ping Mikrotik, but test1 VM - does not.

If i chose some 'Primary exit node' (for example - ainz) - everything seems to be ok, unless ainz goes down - then all VMs lose access to external nets, even if nodes, where they are located, still have BGP links to Mikrotik.

So i am lost here and will appreciate any advice, thanks in advance.

Update: nevermind, Mikrotik (even in v7.15) does not support BGP ECMP inside one BGP instance. But ECMP supported on routing table level. Solution: BGP instance defined as having same router-id, so use different router-id on Mikrotik side for each Proxmox node.

Mikrotik route table when all thing working:

[admin@rb] > ip route print where bgp and dst-address in 192.168.100.0/24
Flags: D - DYNAMIC; A - ACTIVE; b - BGP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
     DST-ADDRESS        GATEWAY       DISTANCE
DAb+ 192.168.100.0/24   192.168.6.13        20
DAb+ 192.168.100.0/24   192.168.6.12        20
DAb+ 192.168.100.0/24   192.168.6.11        20
DAb+ 192.168.100.11/32  192.168.6.13        20
DAb+ 192.168.100.11/32  192.168.6.12        20
DAb+ 192.168.100.12/32  192.168.6.13        20
DAb+ 192.168.100.12/32  192.168.6.11        20

P.S. Still not sure about /32 routes coming from nodes that does not contain particular VM though...

 

[ferrets]

As far as I know, Mikrotik RouterOS(v7.15) does not support EVPN.

 

[Pinkbyte]

As far as I know, Mikrotik RouterOS(v7.15) does not support EVPN.

Proxmox in bgp controller exports EVPN routes as standard ipv4 AFI, not l2vpn, so support from Mikrotik for L2VPN is not needed.

 

[incognitonoobr]

@Pinkbyte you marked this as solved, mind sharing how you solved this?

 

[Pinkbyte]

@incognitonoobr, as i said in Update section of OP:
"nevermind, Mikrotik (even in v7.15) does not support BGP ECMP inside one BGP instance. But ECMP supported on routing table level. Solution: BGP instance defined as having same router-id, so use different router-id on Mikrotik side for each Proxmox node."

 

[incognitonoobr]

i see that now, thanks

do you know if proxmox can be used to replace pfsense entirely for BGP with BYOIP?
i have a thread i created here about it https://forum.proxmox.com/threads/c...lace-pfsense-for-bgp-setup-with-byoip.153157/