Proxmox Cluster SDN EVPN ping destination unreachable

leonidas_o · Nov 21, 2022

I already changed this option "datacenter" to "traditional" a couple of days ago, when I first saw that option but wasn't doing anything. I will try again in a sec.

A ping from node2 to node1:

Code:

64 bytes from 88.XX.XX.XX: icmp_seq=1 ttl=61 time=0.391 ms
64 bytes from 88.XX.XX.XX: icmp_seq=2 ttl=61 time=0.459 ms
64 bytes from 88.XX.XX.XX: icmp_seq=3 ttl=61 time=0.477 ms

I don't know, but this has to be enough, even the proxmox cluster is running with this.
Do you see anything else in the logs?

Is it possible, that bgp or whatever uses ipv6 for something and if it isn't configured correctly, then it dies silently like this? Otherwise this must be a bug, I mean, today I updated to the latest proxmox packages and it's always the same behaviour. EVPN simply does not work, not out of the box, not at all for me. Do you use it with older Proxmox package versions?

leonidas_o · Nov 26, 2022

Can someone confirm that evpn is working with the latest proxmox package versions?

MelonSorbet · Nov 21, 2023

Hi, did you ever find a solution to this?

Running into the exact same problem, with the same symptoms, running pve 8.0.4

leonidas_o · Nov 21, 2023

@MelonSorbet unfortunately no, sorry. In case you solve it, let us know.

MelonSorbet · Nov 22, 2023

Hey @leonidas_o

Just chiming in to say I have found a fix on my end!
Not sure if this is of any help to you, especially since it's so long ago and you've probably moved on to another solution.

What seemed to be the issue for me was the BGP next hop setup, SDN configures on it's own

By checking vtysh and running show bgp nexthop, It showed me the Next-Hop for the route was malformed

I added a BGP controller on my nodes to peer up to an OPNSense Router, where i announced the needed networks to my nodes.

It seems the solution incorrectly (or maybe that's by design, and I'm thinking about this in a wrong way?) assumes all your EVPN Controller nodes will have a direct L2 link to eachother.

Oh well, I can continue my exam project as planned now.. phew...
Time for some sleep

virgil246 · Jan 22, 2024

Hello everyone,

I hope this message finds you well. I'm currently dealing with an issue that seems to resemble an older thread, particularly the one posted by leonidas_o. Initially, I noticed that the BGP status on two Proxmox VE nodes consistently showed "never" for the "up/down" header,
even though BGP appears to be normal.

However, I am still encountering problems with my EVPN configuration. If anyone has insights into what might be happening between my nodes, I would greatly appreciate your assistance.

Your expertise and suggestions on resolving this matter would be invaluable. Thank you in advance for your help!
If additional information is needed, please mention me.

Currently, CT1 with vnet2 on Node 1 cannot ping CT2 with vnet1 on Node 2

Node 1 192.168.88.10 Node 2 192.168.1.110

Code:

#version:52

auto test
iface test
        bridge_ports vxlan_test
        bridge_stp off
        bridge_fd 0
        mtu 1450

auto vnet1
iface vnet1
        address 10.0.1.1/24
        hwaddress BC:24:11:2D:80:71
        bridge_ports vxlan_vnet1
        bridge_stp off
        bridge_fd 0
        mtu 1450
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vnet2
iface vnet2
        address 10.0.2.1/24
        hwaddress BC:24:11:2D:80:71
        bridge_ports vxlan_vnet2
        bridge_stp off
        bridge_fd 0
        mtu 1450
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.88.10
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_test
iface vxlan_test
        vxlan-id 111
        vxlan_remoteip 192.168.1.110
        mtu 1450

auto vxlan_vnet1
iface vxlan_vnet1
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.88.10
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnet2
iface vxlan_vnet2
        vxlan-id 12000
        vxlan-local-tunnelip 192.168.88.10
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto xvrf_evpnzone
iface xvrf_evpnzone
        link-type veth
        address 10.255.255.1/30
        veth-peer-name xvrfp_evpnzone
        mtu 1500

auto xvrfp_evpnzone
iface xvrfp_evpnzone
        link-type veth
        address 10.255.255.2/30
        veth-peer-name xvrf_evpnzone
        vrf vrf_evpnzone
        mtu 1500

Code:

#version:52

auto test
iface test
        bridge_ports vxlan_test
        bridge_stp off
        bridge_fd 0
        mtu 1450

auto vnet1
iface vnet1
        address 10.0.1.1/24
        hwaddress BC:24:11:2D:80:71
        bridge_ports vxlan_vnet1
        bridge_stp off
        bridge_fd 0
        mtu 1450
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vnet2
iface vnet2
        address 10.0.2.1/24
        hwaddress BC:24:11:2D:80:71
        bridge_ports vxlan_vnet2
        bridge_stp off
        bridge_fd 0
        mtu 1450
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route add vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.1.110
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_test
iface vxlan_test
        vxlan-id 111
        vxlan_remoteip 192.168.88.10
        mtu 1450

auto vxlan_vnet1
iface vxlan_vnet1
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.1.110
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_vnet2
iface vxlan_vnet2
        vxlan-id 12000
        vxlan-local-tunnelip 192.168.1.110
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

Node 1

Bash:

root@pve:~# vtysh -c 'sh bgp summary'

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.88.10, local AS number 65000 vrf-id 0
BGP table version 38
RIB entries 0, using 0 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.88.1    4      65001       675       674        0    0    0 00:33:28            0        0 N/A

Total number of neighbors 1

L2VPN EVPN Summary (VRF default):
BGP router identifier 192.168.88.10, local AS number 65000 vrf-id 0
BGP table version 0
RIB entries 15, using 2880 bytes of memory
Peers 1, using 725 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor                    V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
Kenny-Laptop(192.168.1.110) 4      65000       677       673        0    0    0 00:32:31            5        5 N/A

Total number of neighbors 1

Bash:

root@pve:~# vtysh -c "sh ip bgp l2vpn evpn"
BGP table version is 2, local router ID is 192.168.88.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.110:2
   i[3]:[0]:[32]:[192.168.1.110]
                    192.168.1.110(Kenny-Laptop)
                                                  100      0 i
                    RT:65000:111 ET:8
Route Distinguisher: 192.168.1.110:3
   i[3]:[0]:[32]:[192.168.1.110]
                    192.168.1.110(Kenny-Laptop)
                                                  100      0 i
                    RT:65000:11000 ET:8
Route Distinguisher: 192.168.1.110:4
   i[3]:[0]:[32]:[192.168.1.110]
                    192.168.1.110(Kenny-Laptop)
                                                  100      0 i
                    RT:65000:12000 ET:8
Route Distinguisher: 192.168.1.110:5
   i[5]:[0]:[24]:[10.0.1.0]
                    192.168.1.110(Kenny-Laptop)
                                             0    100      0 ?
                    RT:65000:10000 ET:8 Rmac:6a:cc:3d:82:00:1b
   i[5]:[0]:[24]:[10.0.2.0]
                    192.168.1.110(Kenny-Laptop)
                                             0    100      0 ?
                    RT:65000:10000 ET:8 Rmac:6a:cc:3d:82:00:1b
Route Distinguisher: 192.168.88.10:2
 *> [3]:[0]:[32]:[192.168.88.10]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:111
Route Distinguisher: 192.168.88.10:3
 *> [3]:[0]:[32]:[192.168.88.10]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:11000
Route Distinguisher: 192.168.88.10:4
 *> [2]:[0]:[48]:[bc:24:11:93:d5:7a]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:12000
 *> [2]:[0]:[48]:[bc:24:11:93:d5:7a]:[32]:[10.0.2.100]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:12000 RT:65000:10000 Rmac:82:bb:e4:e9:a6:08
 *> [2]:[0]:[48]:[bc:24:11:93:d5:7a]:[128]:[fe80::be24:11ff:fe93:d57a]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:12000
 *> [3]:[0]:[32]:[192.168.88.10]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:12000
Route Distinguisher: 192.168.88.10:5
 *> [5]:[0]:[0]:[0.0.0.0]
                    192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:10000 Rmac:82:bb:e4:e9:a6:08
 *> [5]:[0]:[0]:[::] 192.168.88.10(pve)
                                                       32768 i
                    ET:8 RT:65000:10000 Rmac:82:bb:e4:e9:a6:08

Displayed 13 out of 13 total prefixes

spirit · Jan 22, 2024

@virgil246 can you share your /etc/pve/sdn/*.cfg config ?

They are a bug currently with multiple exit-node, so if you use it, only enable 1 for now. (it's fixed in git master, but not packaged yet).
Also, verify than you don't have any firewall blocking vxlan port (udp/4789)

I'm seeing " xvrfp_". , I think you have enable "local node routing" option on zone, you don't really need this, until you want to reach your vm ip from your proxmox management ip. (It's not buggy, but for debugging it's more easy to not have all options enabled)

If you can't communicate between 2 vms, disable all exit-nodes for now as you don't need them.

virgil246 · Jan 23, 2024

spirit said:
@virgil246 can you share your /etc/pve/sdn/*.cfg config ?

They are a bug currently with multiple exit-node, so if you use it, only enable 1 for now. (it's fixed in git master, but not packaged yet).
Also, verify than you don't have any firewall blocking vxlan port (udp/4789)

I'm seeing " xvrfp_". , I think you have enable "local node routing" option on zone, you don't really need this, until you want to reach your vm ip from your proxmox management ip. (It's not buggy, but for debugging it's more easy to not have all options enabled)

If you can't communicate between 2 vms, disable all exit-nodes for now as you don't need them.

Here is all my conf
CTs' ping is ok when I use VXLAN SDN does it means there is no rule blocking vxlan port?
I still add rule on mikrotik firewall like this (I tried with port 4789 and i don't see any packet through)
about exit node, I remove it from webinterface but I cannot remove primary exit Node
It's OK?

Thanks for your help!

controllers.cfg

Code:

root@pve:~# cat /etc/pve/sdn/controllers.cfg
bgp: bgppve
        asn 65000
        node pve
        peers 192.168.88.1
        bgp-multipath-as-path-relax 0
        ebgp 1

evpn: evpnctl
        asn 65000
        peers 192.168.88.10,192.168.1.110

.running-config

JSON:

{
   "version":53,
   "controllers":{
      "ids":{
         "bgppve":{
            "type":"bgp",
            "ebgp":1,
            "node":"pve",
            "bgp-multipath-as-path-relax":0,
            "asn":65000,
            "peers":"192.168.88.1"
         },
         "evpnctl":{
            "type":"evpn",
            "asn":65000,
            "peers":"192.168.88.10,192.168.1.110"
         }
      }
   },
   "zones":{
      "ids":{
         "test":{
            "peers":"192.168.88.10,192.168.1.110",
            "ipam":"pve",
            "type":"vxlan"
         },
         "evpnzone":{
            "ipam":"pve",
            "vrf-vxlan":10000,
            "type":"evpn",
            "exitnodes-primary":"pve",
            "mac":"BC:24:11:2D:80:71",
            "controller":"evpnctl"
         }
      }
   },
   "subnets":{
      "ids":{
         "test-192.168.200.0-24":{
            "vnet":"test",
            "dhcp-range":[
               "start-address=192.168.200.10,end-address=192.168.200.200"
            ],
            "type":"subnet",
            "gateway":"192.168.200.1"
         },
         "evpnzone-10.0.1.0-24":{
            "vnet":"vnet1",
            "type":"subnet",
            "gateway":"10.0.1.1"
         },
         "evpnzone-10.0.2.0-24":{
            "vnet":"vnet2",
            "type":"subnet",
            "gateway":"10.0.2.1"
         }
      }
   },
   "vnets":{
      "ids":{
         "vnet1":{
            "tag":11000,
            "zone":"evpnzone",
            "type":"vnet"
         },
         "vnet2":{
            "type":"vnet",
            "zone":"evpnzone",
            "tag":12000
         },
         "test":{
            "tag":111,
            "type":"vnet",
            "zone":"test"
         }
      }
   }
}

subnet.cfg

Code:

subnet: test-192.168.200.0-24
        vnet test
        dhcp-range start-address=192.168.200.10,end-address=192.168.200.200
        gateway 192.168.200.1

subnet: evpnzone-10.0.1.0-24
        vnet vnet1
        gateway 10.0.1.1

subnet: evpnzone-10.0.2.0-24
        vnet vnet2
        gateway 10.0.2.1

vnets.cfg

Code:

root@pve:~# cat /etc/pve/sdn/vnets.cfg
vnet: test
        zone test
        tag 111

vnet: vnet1
        zone evpnzone
        tag 11000

vnet: vnet2
        zone evpnzone
        tag 12000

zones.cfg

Code:

root@pve:~# cat /etc/pve/sdn/zones.cfg
vxlan: test
        peers 192.168.88.10,192.168.1.110
        ipam pve

evpn: evpnzone
        controller evpnctl
        vrf-vxlan 10000
        exitnodes-primary pve
        ipam pve
        mac BC:24:11:2D:80:71

spirit · Jan 23, 2024

your config looks fine. (don't worry about exit-node primary, it's not used if not exit-node is defined. I need to fix this gui bug.).

CTs' ping is ok when I use VXLAN SDN does it means there is no rule blocking vxlan port?

yes, if it's working with vxlan zone, this is the same with evpn. (the difference is the controlplane running in bgp)

what is your 2 vm/ct ip address && mac address ?

virgil246 · Jan 23, 2024

spirit said:
your config looks fine. (don't worry about exit-node primary, it's not used if not exit-node is defined. I need to fix this gui bug.).

yes, if it's working with vxlan zone, this is the same with evpn. (the difference is the controlplane running in bgp)

what is your 2 vm/ct ip address && mac address ?

hmm but I use two CT in same vnet while i use VXLAN ZONE
I tried EVPN with same vnet, still not work.
If I give up of using evpn
using VXLAN with an VM as router for public network access is a good practice?
here is my ct's network conf

spirit · Jan 23, 2024

yes sure, you can use vxlan zone with your own virtual router (with 1 interface in vxlan, and 1 interface in real network).

About your frr "vtysh -c sh bgp l2vpn vpn", this is strange, I correctly see the 10.0.2.100 ip, but not the 10.0.1.100, same for mac adress.

Could be interessing to see result of "vtysh -c "sh ip bgp l2vpn evpn" on other node,
and also "vtysh -c sh bgp summary".

Maybe your nodes are not able to establish the bgp session (tcp/179 for firewall)

virgil246 · Jan 24, 2024

spirit said:
yes sure, you can use vxlan zone with your own virtual router (with 1 interface in vxlan, and 1 interface in real network).

About your frr "vtysh -c sh bgp l2vpn vpn", this is strange, I correctly see the 10.0.2.100 ip, but not the 10.0.1.100, same for mac adress.

Could be interessing to see result of "vtysh -c "sh ip bgp l2vpn evpn" on other node,
and also "vtysh -c sh bgp summary".

Maybe your nodes are not able to establish the bgp session (tcp/179 for firewall)

It seems right. After restarting the FRR services, the status of both nodes changed back to 'never' for 'Up/Down'. I will attempt packet capture to gather further evidence on this issue.

After Packet Capture on both side (tcpdump -i vmbr0 tcp port 179) and a router between it, I didn't see any packet about dst port 179, except that packet i test port with telnet.
I have no idea what happend here

Thank you for all your assistance so far.

JiLPi · Jan 24, 2024

Just a word to mention that we are in a very similar situation.
Trying to connect two nodes with public IP addresses on different networks (they can only reach others through layer 3 - no layer 2 at all)
No firewall, raw Proxmox (which works great otherwise).

We can create the zone using an EVPN controller and the corresponding VNet (10.0.0.0/24)

VMs on the same node can ping each others, and reach the internet if their node appears in the list of "Exit nodes".
But a VM on one node cannot ping a VM on the other node.

We also get "up/down: never" when checking the status on both sides.

We're totally new with BGP and still very much rely on whatever we can find on Internet to debug and try to make progress. So thanks for the discussion so far, and we hope we can contribute to (and maybe benefit from

) it.

spirit · Jan 25, 2024

JiLPi said:
Just a word to mention that we are in a very similar situation.
Trying to connect two nodes with public IP addresses on different networks (they can only reach others through layer 3 - no layer 2 at all)
No firewall, raw Proxmox (which works great otherwise).

We can create the zone using an EVPN controller and the corresponding VNet (10.0.0.0/24)

VMs on the same node can ping each others, and reach the internet if their node appears in the list of "Exit nodes".
But a VM on one node cannot ping a VM on the other node.

We also get "up/down: never" when checking the status on both sides.

We're totally new with BGP and still very much rely on whatever we can find on Internet to debug and try to make progress. So thanks for the discussion so far, and we hope we can contribute to (and maybe benefit from ) it.

up/updown: never mean than bgp connection is not established.
if you don't have firewall, this is stange, or maybe you have wrong ips defined in controller configuration

I need more info to help (/etc/pve/sdn/*.cfg , /etc/frr/frr.conf, /etc/network/interfaces.d/sdn , /etc/network/interfaces) for both nodes.

Edit:

Also verifiy that you correctly have install frr package from proxmox repo

Code:

 dpkg -l|grep frr
ii  frr                                  8.5.2-1+pve1

with this package, bgp should be enable by default

Code:

cat /etc/frr/daemons |grep bgpd
bgpd=yes

Search

Search

Proxmox Cluster SDN EVPN ping destination unreachable

leonidas_o

Member

leonidas_o

Member

MelonSorbet

New Member

leonidas_o

Member

MelonSorbet

New Member

virgil246

New Member

spirit

Distinguished Member

virgil246

New Member

spirit

Distinguished Member

virgil246

New Member

spirit

Distinguished Member

virgil246

New Member

JiLPi

New Member

spirit

Distinguished Member