[SOLVED] Management plan vs VM on overlay

C

cyruspy

Renowned Member
Jul 2, 2013
114
8
83
Hello!,

I'm trying to integrate PVE via OIDC to a Keycloak server. The thing is, the server is running as a VM on top of the same cluster and is a client of a EVPN/VXLAN VNI/Subnet.

Even though the anycast GW is attached to a VRF, the traffic originated from the Management plane seems to exit through the directly attached interface, even though it should be isolated.

Am I missing anything here?

Topology:

PVE Management <-- VLAN --> switch <--VLAN--> vFW <--BGP--> PVE node Exit nodes

Instead of going all the way throught the peering point, the PVE node is taking a shortcut via the local interface.

tcpdump shows the "gateway" trying to reach the web servers,
 
Last edited:
Just checked:

net.ipv4.tcp_l3mdev_accept = 0

The local processes bound to default/global VRF should not touch forward requests through VRF based interfaces.

1- Want: FRR working with VRFs
2- Don't want PVEProxy going out through a VRF interface
 
Anybody?.

Today I found something odd. Having 2 subnets in the overlay, same VRF:

VM1 on subnet1 can reach proxmox web portal and SSH on host1 (routing is working as it should)
- Traffic properly routed when VM initiates de connection?

host1 cannot connect to VM2 running a web service on subnet2
- Traffic not properly routed when initiated by host?

subnet1 & subnet2 are served by PVE overlay (EVPN/VXLAN).
 
Last edited:
Bash: 
root@pve-01:~/bin# ip addr show dev ol111001
191: ol111001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master vrf_L01VPN01 state UP group default qlen 1000
   link/ether bc:24:11:e6:34:58 brd ff:ff:ff:ff:ff:ff
   inet 192.168.111.1/25 scope global ol111001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fee6:3458/64 scope link
      valid_lft forever preferred_lft forever

root@pve-01:~/bin# ip addr show dev ol107001
63: ol107001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master vrf_SDCVPN01 state UP group default qlen 1000
   link/ether bc:24:11:a9:f9:46 brd ff:ff:ff:ff:ff:ff
   inet 192.168.107.1/27 scope global ol107001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fea9:f946/64 scope link
      valid_lft forever preferred_lft forever

root@dev-sdc-pve-01:~/bin# ip route get 192.168.111.10
192.168.111.10 dev ol111001 src 192.168.111.1 uid 0
    cache

root@dev-sdc-pve-01:~/bin# ip route get 192.168.107.10
192.168.107.10 dev ol107001 src 192.168.107.1 uid 0
cache

Bash: 
root@pve-01:~/bin# sysctl  net.ipv4|grep l3
net.ipv4.raw_l3mdev_accept = 0
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.udp_l3mdev_accept = 0

Bash: 
root@pve-01:~/bin# ip rule
1000:   from all lookup [l3mdev-table]
32765:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Configuration:

"ip r" output: https://pastebin.com/Q9sF8uMv
"frr.conf.local" content: https://pastebin.com/KAqNqKB1
rendered "frr.conf": https://pastebin.com/gUpYnuc0
/etc/pve/sdn/*: https://pastebin.com/U7yjNe5N
"/etc/network/interfaces" for pve-01: https://pastebin.com/smEfYUJw
 
Last edited:
Fixed!.

The trick was removing the exit nodes (all of them) from the EVPN zone, since it leaks routes to the global/default table.

That is paired with the manual BGP instances + VRF definition for the interface and traffic flow works as expected.
 
cyruspy said:
Fixed!.

The trick was removing the exit nodes (all of them) from the EVPN zone, since it leaks routes to the global/default table.

That is paired with the manual BGP instances + VRF definition for the interface and traffic flow works as expected.
Click to expand...
Hello, do you mind sharing your configuration? i'm stuck at the same exact moment.
 
vatican_psycho said:
Hello, do you mind sharing your configuration? i'm stuck at the same exact moment.
Click to expand...
Hello, pastebin links still work. Only difference is the exit nodes were deleted.

Will share a proper configuration guide once I get access to that environment again.

Please report back if it works.

Validated with 8.3, pending testing on 8.4 (I hope it doesn't break)
 
cyruspy said:
Hello, pastebin links still work. Only difference is the exit nodes were deleted.

Will share a proper configuration guide once I get access to that environment again.

Please report back if it works.

Validated with 8.3, pending testing on 8.4 (I hope it doesn't break)
Click to expand...
Oh sorry, i missed this.

I made some partial tweaks to my config, and yes, it works partially, i was able to advert routes into SDN, now i'm dealing with outsize.
Thank you. Tested on 8.4.1.

EDIT:
Works like a charm ;)
 
Last edited:
And it not working. Well, BGP peers exchange routes, but then PVE nodes won't communicate with each other (as evpn peers). EPVN peers stops talking as soon as I attached interface to SDN vrf, on both sites (2-node cluster). Another thing, sdn item from server view (navigation bar) does work (throw error related to JSON serialization).

I will try tomorrow again to do some more debugging.

EDIT: After some thinking i redesigned:
  1. separate interface for EVPN and BGP peering (i had both on same int before)
  2. separate vrf for bgp peering on both vyos rotuers (they acts as exit nodes for proxmox nodes)
Everything works as intended. Will post it as some tutorial/guide.
 
Last edited:
And again not working...
cyruspy said:
Hello, pastebin links still work. Only difference is the exit nodes were deleted.

Will share a proper configuration guide once I get access to that environment again.

Please report back if it works.

Validated with 8.3, pending testing on 8.4 (I hope it doesn't break)
Click to expand...
And now I have so strange behavior. Not all ping goes as they should. Could you configuration of your exit nodes (firewall I guess)?
 
I definitely need write a tutorial, I suffered with this.

do you have a diagram?.

I didn't go the Vyos route because I didn't want to maintain yet another component. On the bright side, it can act as EVPN exit node: if you want this, don't use my recipe and instead use exit nodes and route targets to handle VRF separation.

If you are reaching the external world vía regular BGP as I do, exit nodes as defined in the SDN module should not be used/set. My recipe is required if the external nodes (firewall/routers) can handle BGP but not MPBGP/EVPN.

vatican_psycho said:
And again not working...

And now I have so strange behavior. Not all ping goes as they should. Could you configuration of your exit nodes (firewall I guess)?
Click to expand...
 
Last edited:
cyruspy said:
I definitely need write a tutorial, I suffered with this.

do you have a diagram?.

I didn't go the Vyos route because I didn't want to maintain yet another component. On the bright side, it can act as EVPN exit node: if you want this, don't use my recipe and instead use exit nodes and route targets to handle VRF separation.

If you are reaching the external world vía regular BGP as I do, exit nodes as defined in the SDN module should not be used/set. My recipe is required if the external nodes (firewall/routers) can handle BGP but not MPBGP/EVPN.
Click to expand...
This is how it's connected in general.

I had to do the same, because RouterOS does not support EVPN yet and i had some problems with BFD.
I discovered that even with only one Vyos VM, and both PVEs peering with it, which is does independently, it results in "lost" packages.
For example: i have 2 DNS servers on PVE-01, one is AdGuard Home, and the other one is Power DNS (local domain).
if to goes (for example ping) from my PC (beyond edge router) to AdGuard Home, it gets routed to PVE-02 and lost at "vxlan br", but if i ping PowerDNS then it goes back as it should.
So for now, PVE-02 does not talk with anyone (except with PVE-01 via EVPN). PVE-02 sees PVE-01 also as default gateway, so it's not ideal.
 

Attachments

  • pic.png
    pic.png
    93.3 KB · Views: 4
RouterOS can do BGP, you drop Vyos and peer RouterOS to your PVE nodes:
PVE --BGP--> RouterOS
* Requires manual configuration at PVE

The other option is:
PVE --EVPN--> Vyos --BGP--> RouterOS
* Uses configuration form / point&click

When ECMP is in use, you need to disable RPF
 
Last edited:
cyruspy said:
RouterOS can do BGP, you drop Vyos and peer RouterOS to your PVE nodes:
PVE --BGP--> RouterOS
* Requires manual configuration at PVE

The other option is:
PVE --EVPN--> Vyos --BGP--> RouterOS
* Uses configuration form / point&click

When ECMP is in use, you need to disable RPF
Click to expand...
That was the plan at the beginning, but i wanted to keep BGP "contained" inside PVE. I'm planning on expanding my SDN by one node for HA.
If one of the PVE node fails, then i could restore Vyos on 3rd host, while still having one backup Vyos (that will keep critical services available at all times) and keeping only 2 BGP sessions (and not one session per PVE node).
 
Last edited:
vatican_psycho said:
That was the plan at the beginning, but i wanted to keep BGP "contained" inside PVE. I'm planning on expanding my SDN by one node for HA.
If one of the PVE node fails, then i could restore Vyos on 3rd host, while still having one backup Vyos (that will keep critical services available at all times) and keeping only 2 BGP sessions (and not one session per PVE node).
Click to expand...
Well, from where I come from, BGP sessions are free :)

I don't need to migrate a virtual firewall if the other host also handles North/South routing.
 
You must log in or register to reply here.
Top Bottom