EVPN-VXLAN "Transport endpoint is not connected" after node takedown and rejoin SDN

F

freakingObelix

New Member
Mar 11, 2025
15
1
3
Hey! How are you people?
I was testing SDN fault tolerance and after a while struggling to make everything work as I intended, finally it's alive. But thing is that after testing a "node purposefully takedown" to try HA and see how much time it takes to reboot a vm and all of that, the node I've rebooted joined ok but SDN is logging this:

Code: 
Apr 13 22:41:32 host3 bgpd[2806]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 13 22:41:32 host3 bgpd[2806]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 13 22:41:32 host3 bgpd[2806]: [H4B4J-DCW2R][EC 33554455] 10.10.1.12 [Error] bgp_read_packet error: Connection reset by peer

Apr 13 22:41:32 host3 bgpd[2806]: [H4B4J-DCW2R][EC 33554455] 10.10.1.11 [Error] bgp_read_packet error: Connection reset by peer

I'm able to ping all three nodes perfectly, no firewall rules bothering bgp... I don't know.
Thing is that to solve this, I must change the node IP, then adjust evpncontroller new members IP and then reload&apply SDN and after that it works. I'm sure that someone knows what happens here since i'm fairly new to proxmox and its SDN (its marvellous by the way, I'm not looking back).



Code: 
root@host3:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp2s0f0
iface enp2s0f0 inet manual
        mtu 9000
#Phy Dev 1

auto enp2s0f1
iface enp2s0f1 inet manual
        mtu 9000
#Phy Dev 2

auto bond0
iface bond0 inet manual
        bond-slaves enp2s0f0 enp2s0f1
        bond-miimon 100
        bond-mode balance-xor
        bond-xmit-hash-policy layer2+3
        mtu 9000
#Phy Bond XOR Hash2+3

auto bond0.10
iface bond0.10 inet static
        address 10.10.1.13/24
        mtu 2000
#VLAN SDN VLAN

auto vmbr0
iface vmbr0 inet static
        address 16.100.15.35/16
        gateway 16.100.1.2
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 11-4094
        mtu 1500
#Management / OOB - VLAN1

auto brextnet2
iface brextnet2 inet manual
        bridge-ports vlextnet2
        bridge-stp off
        bridge-fd 0
        mtu 1500
#Gateway 2

auto brextnet3
iface brextnet3 inet manual
        bridge-ports vlextnet3
        bridge-stp off
        bridge-fd 0
        mtu 1500
#Gateway 3

auto brextnet5
iface brextnet5 inet manual
        bridge-ports vlextnet5
        bridge-stp off
        bridge-fd 0
        mtu 1500
#Gateway 5

auto brextnet1
iface brextnet1 inet manual
        bridge-ports vlextnet1
        bridge-stp off
        bridge-fd 0
        mtu 1500
#Gateway 1

auto vlextnet2
iface vlextnet2 inet manual
        mtu 1500
        vlan-id 1002
        vlan-raw-device enp2s0f0
#VLAN1002 Gateway 2

auto vlextnet3
iface vlextnet3 inet manual
        mtu 1500
        vlan-id 1003
        vlan-raw-device enp2s0f0
#VLAN1003 Gateway 3

auto vlextnet5
iface vlextnet5 inet manual
        mtu 1500
        vlan-id 1005
        vlan-raw-device enp2s0f0
#VLAN1005 Gateway 5

auto vlextnet1
iface vlextnet1 inet manual
        mtu 1500
        vlan-id 1001
        vlan-raw-device enp2s0f0
#VLAN1001 Gateway 1

source /etc/network/interfaces.d/*

Code: 
root@host3:~# cat /etc/pve/sdn/*.cfg
evpn: evpnctlr
        asn 65001
        peers 10.10.1.11,10.10.1.12,10.10.1.13

subnet: evpn1-192.168.103.0-24
        vnet ar0

subnet: evpn3-10.42.0.0-16
        vnet ho1

subnet: evpn2-10.40.0.0-16
        vnet vm1

subnet: evpn3-172.16.0.0-16
        vnet rr1

subnet: evpn2-10.41.1.0-24
        vnet as1

vnet: ar0
        zone evpn1
        tag 1001

vnet: vm1
        zone evpn2
        tag 1003

vnet: as1
        zone evpn2
        tag 1002

vnet: ho1
        zone evpn3
        tag 1004

vnet: rr1
        zone evpn3
        tag 1005

evpn: evpn1
        controller evpnctlr
        vrf-vxlan 101
        disable-arp-nd-suppression 1
        ipam pve
        mac BC:24:11:13:BD:45
        mtu 1500

evpn: evpn2
        controller evpnctlr
        vrf-vxlan 102
        disable-arp-nd-suppression 1
        ipam pve
        mac BC:24:11:81:CF:70
        mtu 1500

evpn: evpn3
        controller evpnctlr
        vrf-vxlan 103
        disable-arp-nd-suppression 1
        ipam pve
        mac BC:24:11:6E:48:99
        mtu 1500

Core is Mikrotik, Aggregation is a c7000 with bl460c gen9, VC flexfabric 10Gb/24.
I'm using balance-xor because I saw a few problems when enabled LACP but in the papers the flexfabric fully supports LACP.

MAC addresses being messed up maybe? I'm lost here.

Thanks in advance.

Edit: As you can see, no other devices apart from those three nodes are in that vlan, should I set up a router and assign ips using dhcp instead?
 
Last edited:
Do those error messages only occur on startup or are they continously logged? Is this a test cluster, so you could try reproducing the same situation and then send the output of the following commands from the node where it is failing and one from the node where it is succeeding:

Code: 
vtysh -c 'show bgp summary'
vtysh -c 'show bgp neighbor'

The generated FRR config from all nodes would also be interesting:

Code: 
cat /etc/frr/frr.conf
 
Hey thanks for replying!
It is a production cluster. This happens if I take down a host whether cold booting or gracefully rebooting it.

I could however move away all services and reproduce the issue easily. In my next message I will send you both scenarios, connected and after rebooting.
 
Here I am making some tests again. I will try to reproduce that behaviour.

Bash: 
L2VPN EVPN Summary:
BGP router identifier 10.10.1.2, local AS number 65001 VRF default vrf-id 0
BGP table version 0
RIB entries 73, using 9344 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor         V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
host1(10.10.1.1) 4      65001     22986     22796     1003    0    0 18:58:39           15       17 FRRouting/10.2.1
host3(10.10.1.3) 4      65001     23291     22796     1003    0    0 18:58:39           15       17 FRRouting/10.2.1

Total number of neighbors 2
BGP neighbor is 10.10.1.1, remote AS 65001, local AS 65001, internal link
  Local Role: undefined
  Remote Role: undefined
Hostname: host1
 Member of peer-group VTEP for session parameters
  BGP version 4, remote router ID 10.10.1.1, local router ID 10.10.1.2
  BGP state = Established, up for 18:58:39
  Last read 00:00:01, Last write 00:00:01
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured tcp-mss is 0, synced tcp-mss is 1948
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised and received
    AddPath:
      L2VPN EVPN: RX advertised and received
    Paths-Limit:
      L2VPN EVPN: advertised (0) and received (0)
    Dynamic: advertised and received
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received
    Enhanced Route Refresh: advertised and received
    Address Family L2VPN EVPN: advertised and received
    Hostname Capability: advertised (name: host2,domain name: n/a) received (name: host1,domain name: n/a)
    Version Capability: advertised software version (FRRouting/10.2.1) received software version (FRRouting/10.2.1)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: L2VPN EVPN
    End-of-RIB received: L2VPN EVPN
    Local GR Mode: Helper*
    Remote GR Mode: Helper
    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:               22        212
    Keepalives:         22773      22773
    Route Refresh:          0          0
    Capability:             0          0
    Total:              22796      22986
  Minimum time between advertisement runs is 0 seconds

 For address family: L2VPN EVPN
  VTEP peer-group member
  Update group 6, subgroup 13
  Packet Queue length 0
  NEXT_HOP is propagated unchanged to this neighbor
  Community attribute sent to this neighbor(all)
  advertise-all-vni
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *MAP_VTEP_IN
  Route map for outgoing advertisements is *MAP_VTEP_OUT
  15 accepted prefixes

  Connections established 1; dropped 0
  Last reset 18:59:33,  Waiting for peer OPEN (FRRouting/10.2.1)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 10.10.1.2, Local port: 44544
Foreign host: 10.10.1.1, Foreign port: 179
Nexthop: 10.10.1.2
Nexthop global: fe80::121f:74ff:fe35:2f8
Nexthop local: fe80::121f:74ff:fe35:2f8
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Estimated round trip time: 1 ms
Read thread: on  Write thread: on  FD used: 25

  BFD: Type: single hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Up, Last update: 0:18:58:38

BGP neighbor is 10.10.1.3, remote AS 65001, local AS 65001, internal link
  Local Role: undefined
  Remote Role: undefined
Hostname: host3
 Member of peer-group VTEP for session parameters
  BGP version 4, remote router ID 10.10.1.3, local router ID 10.10.1.2
  BGP state = Established, up for 18:58:39
  Last read 00:00:00, Last write 00:00:01
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured tcp-mss is 0, synced tcp-mss is 1948
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised and received
    AddPath:
      L2VPN EVPN: RX advertised and received
    Paths-Limit:
      L2VPN EVPN: advertised (0) and received (0)
    Dynamic: advertised and received
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received
    Enhanced Route Refresh: advertised and received
    Address Family L2VPN EVPN: advertised and received
    Hostname Capability: advertised (name: host2,domain name: n/a) received (name: host3,domain name: n/a)
    Version Capability: advertised software version (FRRouting/10.2.1) received software version (FRRouting/10.2.1)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: L2VPN EVPN
    End-of-RIB received: L2VPN EVPN
    Local GR Mode: Helper*
    Remote GR Mode: Helper
    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:               22        517
    Keepalives:         22773      22773
    Route Refresh:          0          0
    Capability:             0          0
    Total:              22796      23291
  Minimum time between advertisement runs is 0 seconds

 For address family: L2VPN EVPN
  VTEP peer-group member
  Update group 6, subgroup 13
  Packet Queue length 0
  NEXT_HOP is propagated unchanged to this neighbor
  Community attribute sent to this neighbor(all)
  advertise-all-vni
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *MAP_VTEP_IN
  Route map for outgoing advertisements is *MAP_VTEP_OUT
  15 accepted prefixes

  Connections established 1; dropped 0
  Last reset 18:59:33,  No AFI/SAFI activated for peer (FRRouting/10.2.1)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 10.10.1.2, Local port: 179
Foreign host: 10.10.1.3, Foreign port: 34558
Nexthop: 10.10.1.2
Nexthop global: fe80::121f:74ff:fe35:2f8
Nexthop local: fe80::121f:74ff:fe35:2f8
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Estimated round trip time: 0 ms
Read thread: on  Write thread: on  FD used: 33

  BFD: Type: single hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Up, Last update: 0:18:58:37

Now rebooting 'gracefully' the node 2 (10.10.1.2). Note that this is a full mesh setup, direct connection on the aggregation layer and nothing in the middle or behind.
Everything after this line es "after rebooting":

Bash: 
L2VPN EVPN Summary:
BGP router identifier 10.10.1.2, local AS number 65001 VRF default vrf-id 0
BGP table version 0
RIB entries 9, using 1152 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.10.1.1       4      65001         0        27        0    0    0    never       Active        0 N/A
10.10.1.3       4      65001         0        27        0    0    0    never       Active        0 N/A

Total number of neighbors 2
BGP neighbor is 10.10.1.1, remote AS 65001, local AS 65001, internal link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group VTEP for session parameters
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.10.1.2
  BGP state = Active
  Last read 00:04:28, Last write 00:00:04
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                 27          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 27          0
  Minimum time between advertisement runs is 0 seconds

 For address family: L2VPN EVPN
  VTEP peer-group member
  Not part of any update group
  NEXT_HOP is propagated unchanged to this neighbor
  Community attribute sent to this neighbor(all)
  advertise-all-vni
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *MAP_VTEP_IN
  Route map for outgoing advertisements is *MAP_VTEP_OUT
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:04:28,  Waiting for peer OPEN (n/a)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 10.10.1.2, Local port: 58692
Foreign host: 10.10.1.1, Foreign port: 179
Nexthop: 10.10.1.2
Nexthop global: fe80::121f:74ff:fe35:2f8
Nexthop local: fe80::121f:74ff:fe35:2f8
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next connect timer due in 6 seconds
Read thread: off  Write thread: off  FD used: -1

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Unknown, Last update: never

BGP neighbor is 10.10.1.3, remote AS 65001, local AS 65001, internal link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group VTEP for session parameters
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.10.1.2
  BGP state = Active
  Last read 00:04:28, Last write 00:00:04
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                 27          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 27          0
  Minimum time between advertisement runs is 0 seconds

 For address family: L2VPN EVPN
  VTEP peer-group member
  Not part of any update group
  NEXT_HOP is propagated unchanged to this neighbor
  Community attribute sent to this neighbor(all)
  advertise-all-vni
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *MAP_VTEP_IN
  Route map for outgoing advertisements is *MAP_VTEP_OUT
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:04:28,  Waiting for peer OPEN (n/a)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 10.10.1.2, Local port: 36562
Foreign host: 10.10.1.3, Foreign port: 179
Nexthop: 10.10.1.2
Nexthop global: fe80::121f:74ff:fe35:2f8
Nexthop local: fe80::121f:74ff:fe35:2f8
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next connect timer due in 6 seconds
Read thread: off  Write thread: off  FD used: -1

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Unknown, Last update: never

Bash: 
root@host1:~# cat /etc/frr/frr.conf

frr version 8.5.2
frr defaults datacenter
hostname host1
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn1
 vni 101
exit-vrf
!
vrf vrf_evpn2
 vni 102
exit-vrf
!
vrf vrf_evpn3
 vni 103
exit-vrf
!
router bgp 65001
 bgp router-id 10.10.1.1
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65001
 neighbor VTEP bfd
 neighbor 10.10.1.2 peer-group VTEP
 neighbor 10.10.1.3 peer-group VTEP
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65001 vrf vrf_evpn1
 bgp router-id 10.10.1.1
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
router bgp 65001 vrf vrf_evpn2
 bgp router-id 10.10.1.1
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
router bgp 65001 vrf vrf_evpn3
 bgp router-id 10.10.1.1
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
line vty

Bash: 
root@host3:~# cat /etc/frr/frr.conf
frr version 8.5.2
frr defaults datacenter
hostname host3
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn1
 vni 101
exit-vrf
!
vrf vrf_evpn2
 vni 102
exit-vrf
!
vrf vrf_evpn3
 vni 103
exit-vrf
!
router bgp 65001
 bgp router-id 10.10.1.3
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65001
 neighbor VTEP bfd
 neighbor 10.10.1.1 peer-group VTEP
 neighbor 10.10.1.2 peer-group VTEP
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65001 vrf vrf_evpn1
 bgp router-id 10.10.1.3
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
router bgp 65001 vrf vrf_evpn2
 bgp router-id 10.10.1.3
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
router bgp 65001 vrf vrf_evpn3
 bgp router-id 10.10.1.3
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
line vty
 
and the last one, "node2" the one that was rebooted, after coming up.



Bash: 
root@host2:~# cat /etc/frr/frr.conf

frr version 8.5.2

frr defaults datacenter

hostname host2

log syslog informational

service integrated-vtysh-config

!

!

vrf vrf_evpn1

 vni 101

exit-vrf

!

vrf vrf_evpn2

 vni 102

exit-vrf

!

vrf vrf_evpn3

 vni 103

exit-vrf

!

router bgp 65001

 bgp router-id 10.10.1.2

 no bgp hard-administrative-reset

 no bgp default ipv4-unicast

 coalesce-time 1000

 no bgp graceful-restart notification

 neighbor VTEP peer-group

 neighbor VTEP remote-as 65001

 neighbor VTEP bfd

 neighbor 10.10.1.1 peer-group VTEP

 neighbor 10.10.1.3 peer-group VTEP

 !

 address-family l2vpn evpn

  neighbor VTEP activate

  neighbor VTEP route-map MAP_VTEP_IN in

  neighbor VTEP route-map MAP_VTEP_OUT out

  advertise-all-vni

 exit-address-family

exit

!

router bgp 65001 vrf vrf_evpn1

 bgp router-id 10.10.1.2

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

exit

!

router bgp 65001 vrf vrf_evpn2

 bgp router-id 10.10.1.2

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

exit

!

router bgp 65001 vrf vrf_evpn3

 bgp router-id 10.10.1.2

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

exit

!

route-map MAP_VTEP_IN permit 1

exit

!

route-map MAP_VTEP_OUT permit 1

exit

!

line vty



And those awful logs...

Code: 
Apr 14 18:19:21 host2 systemd[1]: Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...

Apr 14 18:19:21 host2 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.

Apr 14 18:19:21 host2 systemd[1]: Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.

Apr 14 18:19:21 host2 systemd[1]: Startup finished in 5.040s (kernel) + 10.240s (userspace) = 15.280s.

Apr 14 18:19:21 host2 zebra[3007]: [WPPMZ-G9797] if_zebra_speed_update: vmbr0 old speed: 0 new speed: 20000

Apr 14 18:19:21 host2 zebra[3007]: [WPPMZ-G9797] if_zebra_speed_update: brextnet2 old speed: 0 new speed: 10000

Apr 14 18:19:21 host2 zebra[3007]: [WPPMZ-G9797] if_zebra_speed_update: brextnet3 old speed: 0 new speed: 10000

Apr 14 18:19:21 host2 zebra[3007]: [WPPMZ-G9797] if_zebra_speed_update: brextnet5 old speed: 0 new speed: 10000

Apr 14 18:19:21 host2 zebra[3007]: [WPPMZ-G9797] if_zebra_speed_update: brextnet1 old speed: 0 new speed: 10000

Apr 14 18:19:22 host2 pve-ha-crm[3757]: status change wait_for_quorum => slave

Apr 14 18:19:29 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:29 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:29 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.1 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:19:29 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.3 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:19:39 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:39 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:39 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.3 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:19:49 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:49 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:49 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.3 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:19:59 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:59 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:19:59 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.1 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:19:59 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.3 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:20:09 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:20:09 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:20:09 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.1 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:20:19 host2 chronyd[3539]: Selected source 162.159.200.123 (2.debian.pool.ntp.org)

Apr 14 18:20:19 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:20:19 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected

Apr 14 18:20:19 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.1 [Error] bgp_read_packet error: Connection reset by peer

Apr 14 18:20:19 host2 bgpd[3021]: [H4B4J-DCW2R][EC 33554455] 10.10.1.3 [Error] bgp_read_packet error: Connection reset by peer



Code: 
root@host2:~# cat /etc/network/interfaces

# network interface settings; autogenerated

# Please do NOT modify this file directly, unless you know what

# you're doing.

#

# If you want to manage parts of the network configuration manually,

# please utilize the 'source' or 'source-directory' directives to do

# so.

# PVE will preserve these directives, but will NOT read its network

# configuration from sourced files, so do not attempt to move any of

# the PVE managed interfaces into external files!

auto lo

iface lo inet loopback

auto enp2s0f0

iface enp2s0f0 inet manual

        mtu 9000

#Phy Dev 1

auto enp2s0f1

iface enp2s0f1 inet manual

        mtu 9000

#Phy Dev 2

auto bond0

iface bond0 inet manual

        bond-slaves enp2s0f0 enp2s0f1

        bond-miimon 100

        bond-mode balance-xor

        bond-xmit-hash-policy layer2+3

        mtu 9000

#Phy Bond XOR Hash2+3

auto bond0.10

iface bond0.10 inet static

        address 10.10.1.2/24

        mtu 2000

#VLAN SDN Bridge

auto vmbr0

iface vmbr0 inet static

        address 1X.XXX.X5.34/16

        gateway 1X.XXX.X.2

        bridge-ports bond0

        bridge-stp off

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 11-4094

        mtu 1500

#Management / OOB - VLAN1

auto brextnet2

iface brextnet2 inet manual

        bridge-ports vlextnet2

        bridge-stp off

        bridge-fd 0

        mtu 1500

#Gateway 2

auto brextnet3

iface brextnet3 inet manual

        bridge-ports vlextnet3

        bridge-stp off

        bridge-fd 0

        mtu 1500

#Gateway 3

auto brextnet5

iface brextnet5 inet manual

        bridge-ports vlextnet5

        bridge-stp off

        bridge-fd 0

        mtu 1500

#Gateway 5

auto brextnet1

iface brextnet1 inet manual

        bridge-ports vlextnet1

        bridge-stp off

        bridge-fd 0

        mtu 1500

#Gateway 1

auto vlextnet2

iface vlextnet2 inet manual

        mtu 1500

        vlan-id 1002

        vlan-raw-device enp2s0f0

#VLAN1002 Gateway 2

auto vlextnet3

iface vlextnet3 inet manual

        mtu 1500

        vlan-id 1003

        vlan-raw-device enp2s0f0

#VLAN1003 Gateway 3

auto vlextnet5

iface vlextnet5 inet manual

        mtu 1500

        vlan-id 1005

        vlan-raw-device enp2s0f0

#VLAN1005 Gateway 5

auto vlextnet1

iface vlextnet1 inet manual

        mtu 1500

        vlan-id 1001

        vlan-raw-device enp2s0f0

#VLAN1001 Gateway 1

source /etc/network/interfaces.d/*

After a while, the node didn't came up by itself so I updated the IP addresses of the vlan10 in every node. Applying SDN changes brings up everything as normal.
Code: 
Apr 14 23:06:12 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected
Apr 14 23:06:12 host2 bgpd[3021]: [TXY0T-CYY6F][EC 100663299] Can't get remote address and port: Transport endpoint is not connected
Apr 14 23:06:12 host2 bgpd[3021]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 10.10.1.10(host1) in vrf default Up
Apr 14 23:06:13 host2 bgpd[3021]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for L2VPN EVPN from 10.10.1.10 in vrf default
Apr 14 23:06:15 host2 pmxcfs[3610]: [status] notice: received log
Apr 14 23:06:20 host2 pmxcfs[3610]: [status] notice: received log
Apr 14 23:06:20 host2 pmxcfs[3610]: [status] notice: received log
Apr 14 23:06:21 host2 bgpd[3021]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 10.10.1.13(host3) in vrf default Up
Apr 14 23:06:22 host2 bgpd[3021]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for L2VPN EVPN from 10.10.1.13 in vrf default
 
Last edited:
Are you on 8.4 with FRR 10.2.1?

Seems like there are issues with BFD sometimes not re-establishing after rebooting. I was able to reproduce this on my cluster and I tried with 10.2.2 and it seems fixed there. Most likely this issue here [1].

You should be able to fix this without changing the SDN configuration by running the following commands on the host that fails to establish connections:

Code: 
$ vtysh
(vtysh) conf t
(vtysh) router bgp 65001
(vtysh) no neighbor VTEP bfd
(vtysh) neighbor VTEP bfd

[1] https://github.com/FRRouting/frr/issues/17751
 
THANK YOU, I will try that inmediately to see if it works. If so, I'd rather prefer to write a script until there is some stability tests, I cannot afford more risk than using the non-registered repo.
 
Confirmed as workaround, works as expected.
I've created a systemd service, triggered one shot when the node starts up.

vi /etc/systemd/system/vtysh-bgp-config.service
with contents:

Bash: 
[Unit]
Description=Configure BGP neighbor bfd via vtysh
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/vtysh-bgp-config.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

and vi /usr/local/bin/vtysh-bgp-config.sh
with contents:

Bash: 
#!/bin/bash

# Wait until vtysh is ready (max 60 seconds)
timeout=60
elapsed=0
interval=5

while ! vtysh -c "show version" &>/dev/null; do
    if [ $elapsed -ge $timeout ]; then
        echo "vtysh did not become ready within $timeout seconds"
        exit 1
    fi
    echo "Waiting for vtysh to be ready..."
    sleep $interval
    elapsed=$((elapsed + interval))
done

# Wait before running commands to ensure FRR is ready
# Note: executing those commands inmediately produced no results, thus
# waiting an extra 30sec to be sure.
sleep 30

# Run commands after vtysh is ready
vtysh -c "conf t" \
      -c "router bgp 65001" \
      -c "no neighbor VTEP bfd" \
      -c "neighbor VTEP bfd"

I've inline commented to be clear, although is absolutely simple.

After all that, just systemctl enable --now vtysh-bgp-config.service did the trick.

Thank you again. I'll wait for the next upgrade of FRR.
 
You must log in or register to reply here.
Top Bottom