EVPN/VXLAN between nodes on different public networks

[JiLPi]

Hi,

We're trying to create a cluster of Proxmox nodes directly connected to Internet (public IP, inexpensive servers provided by OVH). No LAN.
We have 2 nodes atm, one in France (n1) and one in Germany (n2).

We're inexperienced with SDN.

Our objectives are :
- put all VMs on the same VNet / Subnet on any node
- VMs should be able to reach all other VMs on the same VNet regardless of which node they are actually located.
- VMs should be able to access the Internet, and be reachable from the Internet if we set the right forwarding rules.
- Encrypted traffic between nodes

We have disabled all firewalls, at the datacenter level and node levels.
Ports 179/TCP and 4789/UDP can be reached from n1 to n2 and vice versa. Can talk with bgpd on remote nodes with netcat.


1/ VXLAN
We have already successfully established a VXLAN that span accross both nodes. VMs can all ping each other, great. VXLAN traffic between nodes is encrypted using strongswan (https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#_vxlan_ipsec_encryption)

But we have failed to let them reach other networks / Internet. We could not figure out how to define a gateway and let anything exit or enter the VXLAN... Could someone point us to some helpful resources?


2/ EVPN
The possibility to manage different VXLAN sounds interesting; we also tried with EVPN.
It seems to be very straightforward in a "layer 2" environment. But we can't get it to work in our case (through Internet / public networks).

Good:
- VMs can reach other VMs on the same node
- VMs can reach the internet

Issues:
- VMs on one node cannot reach VMs on the other node
- It seems that no traffic transits between nodes
- We don't understand how bgp works in this context, and can't see anything happening (vtxsh# show bgp summary shows that the BGP connexions remains in "Active" state with no information being exchanged between nodes)


Configuration on both nodes follows.

Thanks in advance.

 

[JiLPi]

Here's the configuration on node-1 (145.239.xxx.xxx)

dpkg -l|grep frr
ii  frr                                  8.5.2-1+pve1                        amd64        FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)
ii  frr-pythontools                      8.5.2-1+pve1                        all          FRRouting suite - Python tools

cat /etc/frr/daemons |grep bgpd
bgpd=yes
bgpd_options="   -A 127.0.0.1"
# bgpd_wrap="/usr/bin/daemonize /usr/bin/mywrapper"

cat /etc/pve/sdn/*.cfg
evpn: evpnctr1
        asn 65432
        peers 145.239.yyy.yyy,145.239.xxx.xxx

bgp: bgpnode-1
        asn 65001
        node node-1
        peers 145.239.yyy.yyy
        bgp-multipath-as-path-relax 0
        ebgp 1
        ebgp-multihop 20

bgp: bgpnode-2
        asn 65002
        node node-2
        peers 145.239.xxx.xxx
        bgp-multipath-as-path-relax 0
        ebgp 1
        ebgp-multihop 20

subnet: znVXL01-10.0.3.0-24
        vnet vnVXL01
        gateway 10.0.3.1
        snat 1

subnet: znEVPN01-10.0.10.0-24
        vnet vnEVPN01
        gateway 10.0.10.1
        snat 1

vnet: vnVXL01
        zone znVXL01
        alias VNet VXLAN 01
        tag 200

vnet: vnEVPN01
        zone znEVPN01
        alias VNet EVPN 01
        tag 11000

vxlan: znVXL01
        peers 145.239.xxx.xxx,145.239.yyy.yyy
        ipam pve
        mtu 1370

evpn: znEVPN01
        controller evpnctr1
        vrf-vxlan 10000
        exitnodes node-2,node-1
        exitnodes-local-routing 1
        exitnodes-primary node-1
        ipam pve
        mac BC:24:11:24:54:14

cat /etc/frr/frr.conf
frr version 8.5.1
frr defaults datacenter
hostname node-1
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_znEVPN01
 vni 10000
 ip route 10.0.3.0/24 null0
exit-vrf
!
router bgp 65001
 bgp router-id 145.239.xxx.xxx
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 20
 neighbor 145.239.yyy.yyy peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 145.239.yyy.yyy peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65432
 exit-address-family
exit
!
router bgp 65001 vrf vrf_znEVPN01
 bgp router-id 145.239.xxx.xxx
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family l2vpn evpn
  route-target import 65432:10000
  route-target export 65432:10000
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 10000
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
ip route 10.0.10.0/24 10.255.255.2 xvrf_znEVPN01
!
line vty

cat /etc/network/interfaces.d/sdn
#version:33

auto vnEVPN01
iface vnEVPN01
        address 10.0.10.1/24
        post-up iptables -t nat -A POSTROUTING -s '10.0.10.0/24' -o vmbr0 -j SNAT --to-source 145.239.xxx.xxx
        post-down iptables -t nat -D POSTROUTING -s '10.0.10.0/24' -o vmbr0 -j SNAT --to-source 145.239.xxx.xxx
        post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
        hwaddress BC:24:11:24:54:14
        bridge_ports vxlan_vnEVPN01
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias VNet EVPN 01
        ip-forward on
        arp-accept on
        vrf vrf_znEVPN01

auto vnVXL01
iface vnVXL01
        bridge_ports vxlan_vnVXL01
        bridge_stp off
        bridge_fd 0
        mtu 1370
        alias VNet VXLAN 01

auto vrf_znEVPN01
iface vrf_znEVPN01
        vrf-table auto
        post-up ip route del vrf vrf_znEVPN01 unreachable default metric 4278198272

auto vrfbr_znEVPN01
iface vrfbr_znEVPN01
        bridge-ports vrfvx_znEVPN01
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_znEVPN01

auto vrfvx_znEVPN01
iface vrfvx_znEVPN01
        vxlan-id 10000
        vxlan-local-tunnelip 145.239.xxx.xxx
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnEVPN01
iface vxlan_vnEVPN01
        vxlan-id 11000
        vxlan-local-tunnelip 145.239.xxx.xxx
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnVXL01
iface vxlan_vnVXL01
        vxlan-id 200
        vxlan_remoteip 145.239.yyy.yyy
        mtu 1370

auto xvrf_znEVPN01
iface xvrf_znEVPN01
        link-type veth
        address 10.255.255.1/30
        veth-peer-name xvrfp_znEVPN01
        mtu 1500

auto xvrfp_znEVPN01
iface xvrfp_znEVPN01
        link-type veth
        address 10.255.255.2/30
        veth-peer-name xvrf_znEVPN01
        vrf vrf_znEVPN01
        mtu 1500

cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 145.239.xxx.xxx/24
        gateway 145.239.xxx.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        hwaddress A4:BF:01:2B:33:E9

iface vmbr0 inet6 static
        address 2001:41d0:XXX:XXXX::/64
        gateway 2001:41d0:XXX:XXff:ff:ff:ff:ff


source /etc/network/interfaces.d/*

 

[JiLPi]

Configuration on node-2 (145.239.xxx.xxx):

dpkg -l|grep frr
ii  frr                                  8.5.2-1+pve1                        amd64        FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)
ii  frr-pythontools                      8.5.2-1+pve1                        all          FRRouting suite - Python tools

cat /etc/frr/daemons |grep bgpd
bgpd=yes
bgpd_options="   -A 127.0.0.1"
# bgpd_wrap="/usr/bin/daemonize /usr/bin/mywrapper"

cat /etc/pve/sdn/*.cfg
evpn: evpnctr1
        asn 65432
        peers 145.239.yyy.yyy,145.239.xxx.xxx

bgp: bgpnode-1
        asn 65001
        node node-1
        peers 145.239.yyy.yyy
        bgp-multipath-as-path-relax 0
        ebgp 1
        ebgp-multihop 20

bgp: bgpnode-2
        asn 65002
        node node-2
        peers 145.239.xxx.xxx
        bgp-multipath-as-path-relax 0
        ebgp 1
        ebgp-multihop 20

subnet: znVXL01-10.0.3.0-24
        vnet vnVXL01
        gateway 10.0.3.1
        snat 1

subnet: znEVPN01-10.0.10.0-24
        vnet vnEVPN01
        gateway 10.0.10.1
        snat 1

vnet: vnVXL01
        zone znVXL01
        alias VNet VXLAN 01
        tag 200

vnet: vnEVPN01
        zone znEVPN01
        alias VNet EVPN 01
        tag 11000

vxlan: znVXL01
        peers 145.239.xxx.xxx,145.239.yyy.yyy
        ipam pve
        mtu 1370

evpn: znEVPN01
        controller evpnctr1
        vrf-vxlan 10000
        exitnodes node-2,node-1
        exitnodes-local-routing 1
        exitnodes-primary node-1
        ipam pve
        mac BC:24:11:24:54:14

cat /etc/frr/frr.conf
frr version 8.5.1
frr defaults datacenter
hostname node-2
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_znEVPN01
 vni 10000
 ip route 10.0.3.0/24 null0
exit-vrf
!
router bgp 65002
 bgp router-id 145.239.yyy.yyy
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 20
 neighbor 145.239.xxx.xxx peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 145.239.xxx.xxx peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65432
 exit-address-family
exit
!
router bgp 65002 vrf vrf_znEVPN01
 bgp router-id 145.239.yyy.yyy
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family l2vpn evpn
  route-target import 65432:10000
  route-target export 65432:10000
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn vni 10000
 match evpn route-type prefix
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
ip route 10.0.10.0/24 10.255.255.2 xvrf_znEVPN01
!
line vty
!

cat /etc/network/interfaces.d/sdn
#version:33

auto vnEVPN01
iface vnEVPN01
        address 10.0.10.1/24
        post-up iptables -t nat -A POSTROUTING -s '10.0.10.0/24' -o vmbr0 -j SNAT --to-source 145.239.xxx.xxx
        post-down iptables -t nat -D POSTROUTING -s '10.0.10.0/24' -o vmbr0 -j SNAT --to-source 145.239.xxx.xxx
        post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
        hwaddress BC:24:11:24:54:14
        bridge_ports vxlan_vnEVPN01
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias VNet EVPN 01
        ip-forward on
        arp-accept on
        vrf vrf_znEVPN01

auto vnVXL01
iface vnVXL01
        bridge_ports vxlan_vnVXL01
        bridge_stp off
        bridge_fd 0
        mtu 1370
        alias VNet VXLAN 01

auto vrf_znEVPN01
iface vrf_znEVPN01
        vrf-table auto
        post-up ip route del vrf vrf_znEVPN01 unreachable default metric 4278198272

auto vrfbr_znEVPN01
iface vrfbr_znEVPN01
        bridge-ports vrfvx_znEVPN01
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_znEVPN01

auto vrfvx_znEVPN01
iface vrfvx_znEVPN01
        vxlan-id 10000
        vxlan-local-tunnelip 145.239.xxx.xxx
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnEVPN01
iface vxlan_vnEVPN01
        vxlan-id 11000
        vxlan-local-tunnelip 145.239.xxx.xxx
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnVXL01
iface vxlan_vnVXL01
        vxlan-id 200
        vxlan_remoteip 145.239.yyy.yyy
        mtu 1370

auto xvrf_znEVPN01
iface xvrf_znEVPN01
        link-type veth
        address 10.255.255.1/30
        veth-peer-name xvrfp_znEVPN01
        mtu 1500

auto xvrfp_znEVPN01
iface xvrfp_znEVPN01
        link-type veth
        address 10.255.255.2/30
        veth-peer-name xvrf_znEVPN01
        vrf vrf_znEVPN01
        mtu 1500

cat /etc/network/interfacess
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 145.239.yyy.yyy/24
        gateway 145.239.yyy.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        hwaddress A4:BF:01:1F:46:3A

iface vmbr0 inet6 static
        address 2001:41d0:YYYY:YYYY::/64
        gateway 2001:41d0:YYYY:YYff:ff:ff:ff:ff


source /etc/network/interfaces.d/*

 

[JiLPi]

Other debug information

BGP (similar output on node 1 and node 2)

node-2# show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 145.239.yyy.yyy, local AS number 65002 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
145.239.xxx.xxx   4          0         0         0        0    0    0    never       Active        0 N/A

Total number of neighbors 1

vtysh -c "sh ip bgp l2vpn evpn"
BGP table version is 11, local router ID is 145.239.yyy.yyy
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 145.239.yyy.yyy:2
 *> [3]:[0]:[32]:[145.239.yyy.yyy]
                    145.239.yyy.yyy(node-2)
                                                       32768 i
                    ET:8 RT:65432:200
Route Distinguisher: 145.239.yyy.yyy:3
 *> [3]:[0]:[32]:[145.239.yyy.yyy]
                    145.239.yyy.yyy(node-2)
                                                       32768 i
                    ET:8 RT:65432:11000
Route Distinguisher: 145.239.yyy.yyy:4
 *> [5]:[0]:[0]:[0.0.0.0]
                    145.239.yyy.yyy(node-2)
                                                       32768 i
                    ET:8 RT:65432:10000 Rmac:8e:6d:94:6e:3e:fd
 *> [5]:[0]:[0]:[::] 145.239.yyy.yyy(node-2)
                                                       32768 i
                    ET:8 RT:65432:10000 Rmac:8e:6d:94:6e:3e:fd

Displayed 4 out of 4 total prefixes

iptables (node-1: VXLAN input, node-2 : VXLAN output)

node-2#
root@node-2:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A OUTPUT -s 145.239.yyy.yyy/32 -d 145.239.xxx.xxx/32 -o vmbr0 -p udp -m policy --dir out --pol ipsec --reqid 1 --proto esp -m udp --dport 4789 -j ACCEPT

 

[spirit]

Hi,
begin by something simple, with same asn and without exit-node

evpn: evpnctr1
        asn 65432
        peers 145.239.yyy.yyy,145.239.xxx.xxx

evpn: znEVPN01
        controller evpnctr1
        vrf-vxlan 10000
        ipam pve
        mac BC:24:11:24:54:14

if your nodes are able to communicate on the bgp port + vxlan port, it's should be enough to evpn working in the same asn (ibgp)

you can add later extra bgp controller for each node, if you want to do ebgp (a different asn by node), or if you want to peer with classic bgp to another routers. (I'm not sure that in the ovh context, it's something usefull in your case)


you should see connection established in "show bgp summary", then you should se evpn routes "sh ip bgp l2evpn evpn".
if the vxlan port is correctly open in firewall, the vms should be able to communicate in private.

 

[LightCrimson]

Hello,
Thanks a lot for your help.
I'm working with JiLPi, and we made it as simple as possible :

 cat /etc/pve/sdn/*
evpn: evpnctr1
        asn 65432
        peers 145.239.yyy.yyy,145.239.xxx.xxx

subnet: znEVPN01-10.0.10.0-24
        vnet vnEVPN01
        gateway 10.0.10.1
        snat 1

vnet: vnEVPN01
        zone znEVPN01
        alias VNet EVPN 01
        tag 11000

evpn: znEVPN01
        controller evpnctr1
        vrf-vxlan 10000
        ipam pve
        mac BC:24:11:24:54:14

CT 103 and 105 are on node 1
CT 104 and 106 are on node 2

We verified that we can reach the BGP port from each node :

nc 145.239.yyy.yyy -t bgp
b       EFFAEF

Both of them show that we can establish a remote tcp connection on the bgp port with the remote node.


We then tried to ping from 2 CT in the SAME node (CT 105 to CT 103):

ping 10.0.10.103
PING 10.0.10.103 (10.0.10.103) 56(84) bytes of data.
64 bytes from 10.0.10.103: icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from 10.0.10.103: icmp_seq=2 ttl=64 time=0.045 ms

It works as expected.
We can also ping the gateway (10.0.10.1) on both nodes from their local CTs.


However, when we try to ping a CT to another node's CT (CT 105 to CT 104 for example), it doesn't work:

ping 10.0.10.104
PING 10.0.10.104 (10.0.10.104) 56(84) bytes of data.
From 10.0.10.105 icmp_seq=1 Destination Host Unreachable
From 10.0.10.105 icmp_seq=2 Destination Host Unreachable
From 10.0.10.105 icmp_seq=3 Destination Host Unreachable

Here is also the result from the vtysh -c "show bgp summary" command

vtysh -c "show bgp summary"

L2VPN EVPN Summary (VRF default):
BGP router identifier 145.239.yyy.yyy, local AS number 65432 vrf-id 0
BGP table version 0
RIB entries 5, using 960 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor          V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
145.239.xxx.xxx   4      65432         0         0        0    0    0    never       Active        0 N/A

Total number of neighbors 1

Similarly to @virgil246 in this thread, we don't see anything happening on the bgp port using tcpdump (except when we open a connection ourselves with nc).
https://forum.proxmox.com/threads/p...ination-unreachable.118153/page-2#post-627507


It seems that BGP won't even try to communicate with other peers.

Could the fact that the peers are on another network have any influence here?

 

[pille99]

i did it slightly different
vxlan on the site of the VMs (that i can move the VMs around). the vxlan is connected on a opnsense firewall.

 

[JiLPi]

i did it slightly different
vxlan on the site of the VMs (that i can move the VMs around). the vxlan is connected on a opnsense firewall.

Our setup is totally different.
The nodes are directly connected to the internet and have a public IP.
No open sense firewall, switch, etc.
100% IP / layer 3.

 

[nosydev]

Our setup is totally different.
The nodes are directly connected to the internet and have a public IP.
No open sense firewall, switch, etc.
100% IP / layer 3.

Hi,

I'm in a similar situation (2 nodes with only one interface, directly connected to a public IP, its two OVH bare metal servers), and I have the same issue with BGP peers not connecting to each other.

Did you find a solution ?

 

[JiLPi]

Hi,

I'm in a similar situation (2 nodes with only one interface, directly connected to a public IP, its two OVH bare metal servers), and I have the same issue with BGP peers not connecting to each other.

Did you find a solution ?

Hi,
Unfortunately no, and coincidentally, I've tried again yesterday until 2AM (!) without any success.
Reinstalled the entire cluster from scratch. Follow all instructions that we can find here (thanks @spirit) and on the web. Lots of chatgpt as well. All config files look good. But to no avail. It's crazy.

If you find any way to make this work, please let us know

 

[nosydev]

Hi,

I think a have find a hint for this problem.
the commande

vtysh -c "show ip nht"
51.xx.xx.xx
 unresolved
 Client list: bgp(fd 17)

show that host2 can't resolve ip from host1. Host1 and host2 are not on the same network (public ip..), maybe it's the reason we have this problem ?

this issue https://github.com/FRRouting/frr/issues/16101#issuecomment-2151492816 propose to add "no ip nht resolve-via-default" but I'm not sure how to add it properly ?

Maybe we can also add a static route between node ? I'm totally new to frr / bgp so not sure how to move from here

 

[spirit]

Hi,

I think a have find a hint for this problem.
the commande

vtysh -c "show ip nht"
51.xx.xx.xx
 unresolved
 Client list: bgp(fd 17)

show that host2 can't resolve ip from host1. Host1 and host2 are not on the same network (public ip..), maybe it's the reason we have this problem ?

this issue https://github.com/FRRouting/frr/issues/16101#issuecomment-2151492816 propose to add "no ip nht resolve-via-default" but I'm not sure how to add it properly ?

Maybe we can also add a static route between node ? I'm totally new to frr / bgp so not sure how to move from here

I'm using it in production, on differents subnets (but private network).

how much latency do you have between your hosts through the public network ? (just do a ping).

Then, you need to check with "vtysh -c "show bgp summary" that bgp is established and routes are exchanged. (nodes just need to be able to talk through bgp on port tcp/179).

Then, if bgp is ok, nodes need to be able to communites though vxlan port udp/4789.


if you want to test "no ip nht resolve-via-default"",

just do

#vtysh
#conf t
#no ip nht resolve-via-default

 

[nosydev]

I'm using it in production, on differents subnets (but private network).

how much latency do you have between your hosts through the public network ? (just do a ping).

Then, you need to check with "vtysh -c "show bgp summary" that bgp is established and routes are exchanged. (nodes just need to be able to talk through bgp on port tcp/179).

Then, if bgp is ok, nodes need to be able to communites though vxlan port udp/4789.


if you want to test "no ip nht resolve-via-default"",

just do

#vtysh
#conf t
#no ip nht resolve-via-default

Hi,

for "vtysh -c "show bgp summary", I got the same results that JiLPi, Up/Down is "never"
tcp/179 et udp/4789 (I test with nc -vz -u PUBLIC_IP 4789 and nc -vz PUBLIC_IP 179, result say "open").

I add this (on both nodes) to /etc/frr/frr.conf (and restart frr) :

no ip nht resolve-via-default
!
ip route NEIGHBOR_PUBLIC_IP/24 NODE_PUB_IP
!

and now "vtysh -c "show ip nht"" is resolving now, but ""vtysh -c "show bgp summary"" still show "never" under UP/DOWN


Others tests I did :

show bgp neighbors :

show bgp neighbors
BGP neighbor is 51.xx.xx.Xx, remote AS 65000, local AS 65000, internal link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group BGP for session parameters
  BGP version 4, remote router ID 0.0.0.0, local router ID 51.xx.xx.xx
  BGP state = Active
  Last read 00:00:14, Last write never
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  BGP peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:00:14,  No path to specified Neighbor
  Internal BGP neighbor may be up to 255 hops away.
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 109 seconds
Read thread: off  Write thread: off  FD used: -1

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Unknown, Last update: never

not sure about "No path to specified Neighbor" and how to fix it ?

 

[nosydev]

ho, and latency is about 10ms. both servers are in the same OVH datacenter

 

[nosydev]

I also got a "problem" here :

host001# show ip bgp nexthop
Current BGP nexthop cache:
 HOST2_PUBIP invalid, #paths 0, peer HOST2_PUBIP
  Last update: Tue Aug  6 21:38:07 2024

 

[nosydev]

hi @spirit & @JiLPi,

some updates :

in /etc/frr/frr.conf, adding ip route NEIGHBOR_PUBLIC_IP/32 NODE_PUB_GATEWAY, solved the "unresolve" in "show ip nht", "invalid" in "show ip bgp nexthop" and "No path to specified Neighbor" in "show bgp neighbors"

 

[nosydev]

vtysh -c "show bgp summary" have some change : MsgRcvd and MsgSent counter now increasing (were 0 before), but Up/Down still "never"

host001# show bgp summary


IPv4 Unicast Summary (VRF default):
BGP router identifier NODE_ONE_IP, local AS number 65000 vrf-id 0
BGP table version 1
RIB entries 1, using 192 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 2, using 128 bytes of memory


Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
NODE_2_IP   4      65000         6        12        0    0    0    never         Idle        0 N/A


Total number of neighbors 1

I turn live logs (terminal monitor) and this error show up :

2024-08-07 05:42:31.608 [ERR!] bgpd: [MXM8H-B025K][EC 33554452] NODE_2_IP [Error] Configured AFI/SAFIs do not overlap with received MP capabilities
2024-08-07 05:42:31.608 [ERR!] bgpd: [MVZKX-EG443][EC 33554452] bgp_process_packet: BGP OPEN receipt failed for peer: NODE_2_IP
2024-08-07 05:42:31.699 [ERR!] bgpd: [MXM8H-B025K][EC 33554452] NODE_2_IP [Error] Configured AFI/SAFIs do not overlap with received MP capabilities
2024-08-07 05:42:31.699 [ERR!] bgpd: [MVZKX-EG443][EC 33554452] bgp_process_packet: BGP OPEN receipt failed for peer: NODE_2_IP

 

[nosydev]

Hi,
begin by something simple, with same asn and without exit-node

evpn: evpnctr1
        asn 65432
        peers 145.239.yyy.yyy,145.239.xxx.xxx

evpn: znEVPN01
        controller evpnctr1
        vrf-vxlan 10000
        ipam pve
        mac BC:24:11:24:54:14

if your nodes are able to communicate on the bgp port + vxlan port, it's should be enough to evpn working in the same asn (ibgp)

you can add later extra bgp controller for each node, if you want to do ebgp (a different asn by node), or if you want to peer with classic bgp to another routers. (I'm not sure that in the ovh context, it's something usefull in your case)


you should see connection established in "show bgp summary", then you should se evpn routes "sh ip bgp l2evpn evpn".
if the vxlan port is correctly open in firewall, the vms should be able to communicate in private.

Hi @spirit,

I fix the issue below by completing node2 /etc/frr/frr.conf with the same configuration than node1 (minus the neighbor ip) :

 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  neighbor NODE_ONE_IP activate
  import vrf vrf_zevpn
 exit-address-family

is it normal that frr conf are this different ?

Now my peers are connecting, but VMs not on the same node don't communicate (ping etc..), not sure about the configuration I add

 

[nosydev]

root@host001:~# vtysh -c "show bgp summary"

IPv4 Unicast Summary (VRF default):
BGP router identifier NODE_ONE, local AS number 65000 vrf-id 0
BGP table version 1
RIB entries 1, using 192 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
NODE_TWO        4      65000       707       709        0    0    0 00:26:34            1        1 N/A

Total number of neighbors 1

 

[nosydev]

Hi,
Unfortunately no, and coincidentally, I've tried again yesterday until 2AM (!) without any success.
Reinstalled the entire cluster from scratch. Follow all instructions that we can find here (thanks @spirit) and on the web. Lots of chatgpt as well. All config files look good. But to no avail. It's crazy.

If you find any way to make this work, please let us know

Hi @JiLPi,

I did another test, from scratch (I remove everything first un SDN, "apply", and start from there). I got the internode communication working but, (and I don't understand why because it was working in my previous test), VMs can't communicate with the internet..

Here is my configuration :





AND I add on each node the following lines at the end of /etc/frr/frr.conf + systctl frr restart

no ip nht resolve-via-default
!
ip route Other_node_ip/32 current_node_gateway_ip
!

I really don't understand why the VMs don't have Internet access.. it's driving my crazy...