proxmox 7.0 sdn beta test

Ruffy91 · Jul 27, 2022

Enjoy your holiday

The config as generated now is:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.

spirit · Jul 27, 2022

Ruffy91 said:

Enjoy your holiday

The config as generated now is:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.

Ruffy91 said:

Enjoy your holiday

The config as generated now is:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:

Code:

frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.

Maybe copy "neighbor bgp activate" in vrf router ipv4 unicast section. But keep it too in main section. I ll do test next week.

spirit · Jul 28, 2022

spirit said:
Maybe copy "neighbor bgp activate" in vrf router ipv4 unicast section. But keep it too in main section. I ll do test next week.

Just thinking about it, it ll not work directly in the vrf, until you hace a dedicated ethx interface in this vrf with a dedicated ip.

spirit · Aug 1, 2022

@Ruffy91

I'll do test on my side next week, but I think it can be done with policy based routing (pbr)

https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-41/Layer-3/Policy-based-Routing/
where you can define a differents gatewa for a specific subnet source.

Ruffy91 · Aug 2, 2022

I added a PBR for any traffic on interface evpn01 which added a rule and route table which seem correct, but the rule is ignored by the route selection:
root@chsfl1-cl01-pve01:~# ip rule
300: from all iif evpn01 lookup 10000 proto zebra
1000: from all lookup [l3mdev-table]
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
root@chsfl1-cl01-pve01:~# ip route list table 10000
default nhid 673 via 100.111.64.1 dev vmbr0_164 proto pbr metric 20
root@chsfl1-cl01-pve01:~# ip route get 1.1.1.1 from 10.182.3.100 iif evpn01
1.1.1.1 from 10.182.3.100 via 10.182.2.1 dev vmbr0_182
cache iif evpn01

I think it should even be possible to select the correct vrf with PBR (set vrf VRF-NAME) but this wont help if the rules are ignored.

spirit · Aug 3, 2022

Ruffy91 said:
I added a PBR for any traffic on interface evpn01 which added a rule and route table which seem correct, but the rule is ignored by the route selection:
root@chsfl1-cl01-pve01:~# ip rule
300: from all iif evpn01 lookup 10000 proto zebra
1000: from all lookup [l3mdev-table]
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
root@chsfl1-cl01-pve01:~# ip route list table 10000
default nhid 673 via 100.111.64.1 dev vmbr0_164 proto pbr metric 20
root@chsfl1-cl01-pve01:~# ip route get 1.1.1.1 from 10.182.3.100 iif evpn01
1.1.1.1 from 10.182.3.100 via 10.182.2.1 dev vmbr0_182
cache iif evpn01

I think it should even be possible to select the correct vrf with PBR (set vrf VRF-NAME) but this wont help if the rules are ignored.

thanks for testing. I'll do a lab next week when I'll be back at work.

grimsrue · Aug 30, 2022

I wanted to ask if there is going to be an update to allow IPv6 ebgp connectivity?
As it stands now, trying to add IPv6 peering addresses to the the "Peers" window in the SDN UI will add them to the "BGP" peer group. This will mix the v4 and v6 addresses together and wont allow the IPv6 addresses to establish a peering connection. It would be nice to have a v4 and a separate v6 Peer window in the SDN BGP UI so that v4 peering IPs get their own peer group and v6 peering IPs get their own peer group. The IPv6 peer group will also have to be activated under the "address-family ipv6 unicast"

Thinking that it might look something like this below.
Current configuration:

!
frr version 8.3
frr defaults datacenter
hostname PROX-FRR-NSX-B1
log syslog informational
hostname Prox-FrrRouter-B1
service integrated-vtysh-config
!
vrf vrf_EVPNZone
vni 4000
exit-vrf
!
router bgp 48000
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor BGPv4 peer-group
neighbor BGPv4 remote-as external
neighbor BGPv6 peer-group
neighbor BGPv6 remote-as external
neighbor VTEP peer-group
neighbor VTEP remote-as 48000
neighbor VTEP bfd
neighbor 135.xxx.xxx.8 peer-group BGPv4
neighbor 135.xxx.xxx.9 peer-group BGPv4
neighbor 2001:xxxx:xxxx:xxxx::x:4a peer-group BGPv6
neighbor 2001:xxxx:xxxx:xxxx::x:4b peer-group BGPv6
neighbor 135.xxx.xxx.15 peer-group VTEP
neighbor 135.xxx.xxx.16 peer-group VTEP
neighbor 135.xxx.xxx.17 peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:5b peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6a peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6b peer-group VTEP
!
address-family ipv4 unicast
neighbor BGPv4 activate
neighbor BGPv4 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family ipv6 unicast
neighbor BGPv6 activate
neighbor BGPv6 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router bgp 48000 vrf vrf_EVPNZone
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
end

spirit · Aug 30, 2022

grimsrue said:
I wanted to ask if there is going to be an update to allow IPv6 ebgp connectivity?
As it stands now, trying to add IPv6 peering addresses to the the "Peers" window in the SDN UI will add them to the "BGP" peer group. This will mix the v4 and v6 addresses together and wont allow the IPv6 addresses to establish a peering connection. It would be nice to have a v4 and a separate v6 Peer window in the SDN BGP UI so that v4 peering IPs get their own peer group and v6 peering IPs get their own peer group. The IPv6 peer group will also have to be activated under the "address-family ipv6 unicast"

Thinking that it might look something like this below.
Current configuration:

!
frr version 8.3
frr defaults datacenter
hostname PROX-FRR-NSX-B1
log syslog informational
hostname Prox-FrrRouter-B1
service integrated-vtysh-config
!
vrf vrf_EVPNZone
vni 4000
exit-vrf
!
router bgp 48000
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor BGPv4 peer-group
neighbor BGPv4 remote-as external
neighbor BGPv6 peer-group
neighbor BGPv6 remote-as external
neighbor VTEP peer-group
neighbor VTEP remote-as 48000
neighbor VTEP bfd
neighbor 135.xxx.xxx.8 peer-group BGPv4
neighbor 135.xxx.xxx.9 peer-group BGPv4
neighbor 2001:xxxx:xxxx:xxxx::x:4a peer-group BGPv6
neighbor 2001:xxxx:xxxx:xxxx::x:4b peer-group BGPv6
neighbor 135.xxx.xxx.15 peer-group VTEP
neighbor 135.xxx.xxx.16 peer-group VTEP
neighbor 135.xxx.xxx.17 peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:5b peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6a peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6b peer-group VTEP
!
address-family ipv4 unicast
neighbor BGPv4 activate
neighbor BGPv4 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family ipv6 unicast
neighbor BGPv6 activate
neighbor BGPv6 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router bgp 48000 vrf vrf_EVPNZone
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
end

Hi,
yes, I could adapt code for handling mixing ipv4 && ipv6 peer. (I didn't have thinked about this).
Should be easy to implement, I'll try to send a patched version for testing today or tomorrow.

spirit · Aug 30, 2022

@grimsrue

can you test this patched version:

Code:

wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb
dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.

grimsrue · Aug 30, 2022

spirit said:
@grimsrue

can you test this patched version:

Code:

wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.

Sure! Give me a day or so. Have a busy day today. I'll try to find 30 mins today or tonight to download the patch, install it and test it

grimsrue · Aug 30, 2022

spirit said:
@grimsrue

can you test this patched version:

Code:

wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.

@spirit

I had a little time to download your patch and test it. It looks like it is working well.

One suggestion to the UI window for BGP is to add v4/v6 to the end or beginning of Peers so people know that they should be all IPv4 and IPv6 peers into the same field. Same for EVPN.

spirit · Aug 31, 2022

grimsrue said:
@spirit

I had a little time to download your patch and test it. It looks like it is working well.

One suggestion to the UI window for BGP is to add v4/v6 to the end or beginning of Peers so people know that they should be all IPv4 and IPv6 peers into the same field. Same for EVPN.

View attachment 40529

View attachment 40531

ok thanks for testing ! It'll send patch to pve-devel mailing list to include it in next release. I'look to improve the gui too.

lp_xanclas · Sep 5, 2022

Hi,
Are any docs mentioning how to setup BGP controller (with eBGP) to advertise the subnets on the VNETs to an external router?
I managed to receive external advertised subnets on the proxmox server, from a different AS that is configured on the proxmox BGP controller, but I don't know whow to advertise, on proxmox, the VNETs to that external AS.
Thank you.

spirit · Sep 6, 2022

lp_xanclas said:
Hi,
Are any docs mentioning how to setup BGP controller (with eBGP) to advertise the subnets on the VNETs to an external router?
I managed to receive external advertised subnets on the proxmox server, from a different AS that is configured on the proxmox BGP controller, but I don't know whow to advertise, on proxmox, the VNETs to that external AS.
Thank you.

I really need to write some docs with differents exemples.

you need:
1) enable exit-nodes on the zone options.
2) add an extra bgp controller for each exit-node, where on the peers options, you defines all hypervisors ip (like on the evpn controller) + the ip of your external routers + enabled the ebgp checkbox if your external as is different

lp_xanclas · Sep 6, 2022

Hi @spirit , it worked.
Thank you.

lp_xanclas · Oct 10, 2022

Hello,
I've a setup that has a proxmox server IP in the same subnet that my PC. I have a pfsense and VyOS router as VMs running in the proxmox server. To access my VMs inside any SDN VNET, I forward the traffic to my pfsense that then forwards the traffic to VyOs and then delivers it to the VMs on the proxmox. The main thing here is that I have asymmetric routing due the fact that the proxmox is in the same subnet of my PC. So, the reply for example of SSH traffic will be forwarded to that interface that is in the same subnet of my PC, because proxmox as a route for that subnet as directly connected. Pfsense after a while drops the session as expected. I want to understand if there's a way to have that proxmox interface enabled in the same subnet of my PC, but without interfering with the VMs traffic within the SDN?

All has been set using BGP and I can access to the VMs in proxmox without any problem, but due to asymmetric routing stateful connections are dropped. The Vyos na Pfsense are connected each other by a "transit" subnet and I'm using VLANs for the network interfaces of pfsense and vyos VM. These VLANs are in a different bridge of the bridge used in the proxmox interface on PC subnet.

eset · Oct 27, 2022

spirit said:
sorry I'm currently on holiday, I have only access with my phone with poor connection.

if mikrotic don't support evpn correctly
---------------------------------------------------------
for evpn controller: peers: you need to use all proxmox host ips, to exchange evpn routes.

then, for the exit node(pve), you add the bgp controller with the additionnal mikrotik peer (exactly like on your screenshot)

if mikrotic support evpn correctly (with full symmetric l3vni)
--------------------------------------------------------------------------------------

on epvn controler: peers: define all proxmox hosts ip + mikrotik ip
don't define exit node on zone
configure your mikrotik to announce an evpn type5 route 0.0.0.0/0 + an l3vni vxlan interface.

So basically that is what I got previously. What have I done differently?

Btw
> then, for the exit node(pve), you add the bgp controller with the additionnal mikrotik peer (exactly like on your screenshot)
on my screenshoot there is pve from menu bar . I can't add anything else as you said , additional mikrotik peer.

Basically my setup looks like this:

1. BGP Controller

10.0.1.30 that's MikroTik IP address in VLAN 11. Proxmox Is in the same network connected with MikroTik as DHCP Server 10.0.1.0/27
Proxmox IP in that VLAN is 10.0.1.1

2. EVPN Controller

Reminder: 10.0.1.1 is Proxmox host

3. Vnets

4. Zones

And from VM in 10.0.101.0/24 I can ping Proxmox Host 10.0.1.1

But from my Laptop which is behind Unifi Switch I can't ping VM (10.0.101.1)
Stops exactly on Proxmox

❯ traceroute 10.0.101.1
traceroute to 10.0.101.1 (10.0.101.1), 64 hops max, 52 byte packets
1 192.168.1.254 (192.168.1.254) 2.262 ms 1.714 ms 1.608 ms
2 10.255.253.1 (10.255.253.1) 0.901 ms 0.909 ms 0.878 ms
3 10.0.1.1 (10.0.1.1) 2.999 ms 1.059 ms 1.038 ms

Also proxmox after creating a subnet in Vnets with gateway 10.0.101.254 created interface on the proxmox host itself

root@pve:~# cat /etc/network/interfaces.d/sdn
#version:73

auto test1
iface test1
address 10.0.101.254/24
hwaddress 32:1C:C7:B6:78:C5
bridge_ports vxlan_test1
bridge_stp off
bridge_fd 0
mtu 1450
ip-forward on
arp-accept on
vrf vrf_test1

And I can't ping from proxmox this interface IP which is strange

spirit · Oct 27, 2022

eset said:
So basically that is what I got previously. What have I done differently?

Btw
> then, for the exit node(pve), you add the bgp controller with the additionnal mikrotik peer (exactly like on your screenshot)
on my screenshoot there is pve from menu bar . I can't add anything else as you said , additional mikrotik peer.

View attachment 42636

Basically my setup looks like this:

1. BGP Controller

View attachment 42637

10.0.1.30 that's MikroTik IP address in VLAN 11. Proxmox Is in the same network connected with MikroTik as DHCP Server 10.0.1.0/27
Proxmox IP in that VLAN is 10.0.1.1

yes, sorry, This is the correct conf I mean. "Create a bgp controller, with mikrotik ip as peer, where node is the exit-node. (here pve).
But don't change exit-node on the zone.

eset said:
2. EVPN Controller

View attachment 42638

Reminder: 10.0.1.1 is Proxmox host

3. Vnets

View attachment 42639

4. Zones

View attachment 42640

And from VM in 10.0.101.0/24 I can ping Proxmox Host 10.0.1.1

View attachment 42641

But from my Laptop which is behind Unifi Switch I can't ping VM (10.0.101.1)
Stops exactly on Proxmox

what is your laptop ip/subnet ? Do you have a gateway/router between your laptop ip and ip of the proxmox exit-node ?

if yes, you need to have a route on your gateway, like "ip route add 10.0.101.0/24 via 10.0.1.1"

eset said:
Also proxmox after creating a subnet in Vnets with gateway 10.0.101.254 created interface on the proxmox host itself

eset said:
And I can't ping from proxmox this interface IP which is strange

if you want to ping from proxmox directly, you need to enable "exit-node local routing" option on the zone.
but for security, it's better to keep it off.

eset · Oct 27, 2022

what is your laptop ip/subnet ? Do you have a gateway/router between your laptop ip and ip of the proxmox exit-node ?

@spirit you are very helpful and I don't want to waste your time. I see you are engaged to help me so I really wanna help you out so that we both understand each other better. That's why I prepared some extra stuff to help you out with the network scope, how it looks and how it's configured.
Below a very general screenshot from UniFi Controller on my Raspberry Pi. You see that MikroTik (rt-poz-1) is connected to sw-poz-1.
So answering your question: Yes. I do have a gateway which is exactly my MikroTik router between me and proxmox. More detailed config on second image.

Detailed diagram

Proxmox network config

Bash:

auto lo
iface lo inet loopback

iface eno0 inet manual

iface enp8s0f1 inet manual

auto enp8s0f2
iface enp8s0f2 inet manual

auto enp8s0f3
iface enp8s0f3 inet manual

iface enp8s0f0 inet manual hwaddress 00:1e:67:68:00:ce
iface enp8s0f1 inet manual hwaddress 00:1e:67:68:00:cf
iface enp8s0f2 inet manual hwaddress 00:1e:67:68:00:d0
iface enp8s0f3 inet manual hwaddress 00:1e:67:68:00:d1

auto bond0
iface bond0 inet manual
    bond-slaves enp8s0f2 enp8s0f3
    bond-miimon 100
    bond-mode 802.3ad
    mtu 8000
    bond-lacp-rate    1
#ch3 [ g5,g6 ]

auto vmbr0
iface vmbr0 inet manual
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    mtu 8000

auto vmbr0.11
iface vmbr0.11 inet dhcp
    hwaddress 00:1e:67:00:11:01
#    address 10.0.1.1/30    gateway 10.0.1.30
    mtu 8000
auto vmbr0.110
iface vmbr0.110 inet dhcp
    hwaddress 00:1e:67:01:10:53
    mtu 8000

source /etc/network/interfaces.d/*

And interfaces.d/sdn looks like this

Bash:

#version:83

auto test1
iface test1
    address 10.0.101.254/24
    hwaddress 32:1C:C7:B6:78:C5
    bridge_ports vxlan_test1
    bridge_stp off
    bridge_fd 0
    mtu 1450
    ip-forward on
    arp-accept on
    vrf vrf_test1

auto vrf_test1
iface vrf_test1
    vrf-table auto
    post-up ip route del vrf vrf_test1 unreachable default metric 4278198272

auto vrfbr_test1
iface vrfbr_test1
    bridge-ports vrfvx_test1
    bridge_stp off
    bridge_fd 0
    mtu 1450
    vrf vrf_test1

auto vrfvx_test1
iface vrfvx_test1
    vxlan-id 4041
    vxlan-local-tunnelip 10.0.1.1
    bridge-learning off
    bridge-arp-nd-suppress on
    mtu 1450

auto vxlan_test1
iface vxlan_test1
    vxlan-id 4090
    vxlan-local-tunnelip 10.0.1.1
    bridge-learning off
    bridge-arp-nd-suppress on
    mtu 1450

From previous messages I'm pretty sure something isn't right on Proxmox host itself. MikroTik receives BGP advertisement of 10.0.101.0/24 through 10.0.1.1 and traceroute stops on Proxmox host always. When I send packet from my laptop or VM on Proxmox.
Also ping from VM outside into Internet e.g 1.1.1.1 also doesn't work. Example below.

Routing looks like this

Of course ping to that VM works from Hypervisor which is Proxmox host (10.0.1.1)

Bash:

konrad@pve:~$ ping 10.0.101.1 -c 2
PING 10.0.101.1 (10.0.101.1) 56(84) bytes of data.
64 bytes from 10.0.101.1: icmp_seq=1 ttl=64 time=0.402 ms
64 bytes from 10.0.101.1: icmp_seq=2 ttl=64 time=0.270 ms

--- 10.0.101.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1031ms
rtt min/avg/max/mdev = 0.270/0.336/0.402/0.066 ms

konrad@pve:~$ sudo vtysh -c "sh ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 10.0.1.30, vmbr0.11, 09:07:52
C>* 10.0.1.0/27 is directly connected, vmbr0.11, 09:30:37
C>* 10.0.10.0/24 is directly connected, vmbr0.110, 09:30:37
B>* 10.0.101.0/24 [20/0] is directly connected, test1 (vrf vrf_test1), weight 1, 00:48:51
B>* 192.168.0.0/16 [200/0] via 10.0.1.30, vmbr0.11, weight 1, 09:30:37

And other way also VM => PVE

I tried even to ping from mikrotik which is basically a gateway for proxmox (VLAN 11) to omit unifi switch and the same situation: Timeouts on ping

Code:

[konrad@Mike[1]] > ping 10.0.101.1
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 10.0.101.1                                              timeout
    1 10.0.101.1                                              timeout
    2 10.0.101.1                                              timeout
    3 10.0.101.1                                              timeout

if yes, you need to have a route on your gateway, like "ip route add 10.0.101.0/24 via 10.0.1.1"

I have already resolved that by the BGP controller, remember? It advertise that network to my mikrotik.

Code:

Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  10.0.101.0/24                      10.0.1.1                200

ADb - A - active, D - dynamic, b - bgp
so routing works from BGP Controller.

if you want to ping from proxmox directly, you need to enable "exit-node local routing" option on the zone.
but for security, it's better to keep it off.

Exactly. And btw when it's selected I loose default gateway route entry in routing table on Proxmox host. I was hoping that this might help or be the problem but of course it isn't.
I must say that I'm hopeless. I wanted basically isolate VM/Container networks on my proxmox from my LAN so I can easy use IaC tools with Cloud-Init to manage address management with PowerDNS plugin and from my LAN easily use dns suffixes so I would have a substitute for a cloud environment.

I will be very pleased if you could stay with me on this and try to resolve it. I promise I will do a short video tutorial about that because I see there is none and this can be a very helpful HowTo for others who want to join SDN features.

apollo13 · Oct 28, 2022

We are currently considering reworking our existing setup to a simple SDN setup. I noticed that a VLAN Zone requires a bridge as underlying device. What is the reason for this? Wouldn't a bond or any normal interface do as well -- after all the created VNETs are bridges on their own again. I am asking because the proxmox docs seem to suggest

In general, you should configure the VLAN on the interface with the least abstraction layers between itself and the physical NIC.

and using a vnet in SDN results in:

Code:

auto vlan711
iface vlan711
    bridge_ports vmbr0.711
    bridge_stp off
    bridge_fd 0
    alias INTERN

when it could probably just as simply be

Code:

auto vlan711
iface vlan711
    bridge_ports bond0.711
    bridge_stp off
    bridge_fd 0
    alias INTERN

or am I missing something?

Thank you,
Florian

proxmox 7.0 sdn beta test

Active Member

Distinguished Member

Distinguished Member

Distinguished Member

Active Member

Distinguished Member

New Member

Distinguished Member

Distinguished Member

New Member

New Member

Distinguished Member

Member

Distinguished Member

Member

Member

Well-Known Member

Distinguished Member

Well-Known Member

Well-Known Member