proxmox 7.0 sdn beta test

[eset]

@spirit
I tried to ping from my vmbr0.11 interface which is 10.0.1.1 to 10.0.101.254 (gateway of subnet on VNet)

root@pve:~# ping 10.0.101.254 -I vmbr0.11
PING 10.0.101.254 (10.0.101.254) from 10.0.1.1 vmbr0.11: 56(84) bytes of data.
^C
--- 10.0.101.254 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9193ms

My route all route tables looks like this

root@pve:~# ip route show table all
10.0.101.0/24 dev test1 table vrf_test1 proto kernel scope link src 10.0.101.254
local 10.0.101.254 dev test1 table vrf_test1 proto kernel scope host src 10.0.101.254
broadcast 10.0.101.255 dev test1 table vrf_test1 proto kernel scope link src 10.0.101.254
default via 10.0.1.30 dev vmbr0.11
10.0.1.0/27 dev vmbr0.11 proto kernel scope link src 10.0.1.1
10.0.10.0/24 dev vmbr0.110 proto kernel scope link src 10.0.10.252
10.0.101.0/24 nhid 77 dev test1 proto bgp metric 20
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/16 nhid 42 via 10.0.1.30 dev vmbr0.11 proto bgp metric 20
local 10.0.1.1 dev vmbr0.11 table local proto kernel scope host src 10.0.1.1
broadcast 10.0.1.31 dev vmbr0.11 table local proto kernel scope link src 10.0.1.1
local 10.0.10.252 dev vmbr0.110 table local proto kernel scope host src 10.0.10.252
broadcast 10.0.10.255 dev vmbr0.110 table local proto kernel scope link src 10.0.10.252
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.17.0.1 dev docker0 table local proto kernel scope host src 172.17.0.1
broadcast 172.17.255.255 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown
anycast fe80:: dev vrfbr_test1 table vrf_test1 proto kernel metric 0 pref medium
anycast fe80:: dev test1 table vrf_test1 proto kernel metric 0 pref medium
local fe80::301c:c7ff:feb6:78c5 dev test1 table vrf_test1 proto kernel metric 0 pref medium
local fe80::54d8:13ff:feed:6d1c dev vrfbr_test1 table vrf_test1 proto kernel metric 0 pref medium
fe80::/64 dev vrfbr_test1 table vrf_test1 proto kernel metric 256 pref medium
fe80::/64 dev test1 table vrf_test1 proto kernel metric 256 pref medium
multicast ff00::/8 dev vrfbr_test1 table vrf_test1 proto kernel metric 256 pref medium
multicast ff00::/8 dev test1 table vrf_test1 proto kernel metric 256 pref medium
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0.11 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0.110 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fe80:: dev vmbr0 table local proto kernel metric 0 pref medium
anycast fe80:: dev vmbr0.11 table local proto kernel metric 0 pref medium
anycast fe80:: dev vmbr0.110 table local proto kernel metric 0 pref medium
local fe80::21e:67ff:fe00:1101 dev vmbr0.11 table local proto kernel metric 0 pref medium
local fe80::21e:67ff:fe01:1053 dev vmbr0.110 table local proto kernel metric 0 pref medium
local fe80::21e:67ff:fe68:d0 dev vmbr0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev vmbr0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev vmbr0.11 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev vmbr0.110 table local proto kernel metric 256 pref medium

It seems that something isn't working between communication on two route tables - default and the vrf_test1

root@pve:~# ip route show table vrf_test1
10.0.101.0/24 dev test1 proto kernel scope link src 10.0.101.254
local 10.0.101.254 dev test1 proto kernel scope host src 10.0.101.254
broadcast 10.0.101.255 dev test1 proto kernel scope link src 10.0.101.254

It seems that this is a new issue that wasn't considered.

 

[spirit]

We are currently considering reworking our existing setup to a simple SDN setup. I noticed that a VLAN Zone requires a bridge as underlying device. What is the reason for this? Wouldn't a bond or any normal interface do as well -- after all the created VNETs are bridges on their own again. I am asking because the proxmox docs seem to suggest

This is because physical interfaces can be different on each host. (and some users also use OVS for bonding,...).

 

[spirit]

@eset

can you try to disable exit-node-local-routing option ? (because it's complexify the debug).
You don't need to be able to ping from pve host ip to the vm, to get it working.
Proxmox node should only be able to forward traffic between the default vrf && the zone vrf.

and also, remove docker. (I don't known why you have docker, but it's create bridges && routes, and I want to be sure that we don't have any conflict here).

(and if you can, do a clean reboot to be sure)


and then, send me:

/etc/frr/frr.config
/etc/network/interfaces
/etc/network/interfaces.d/sdn
# ip route show table all
# vtysh -c "sh ip route"
# vtysh -c "sh bgp l2vpn evpn"


(The bgp session with mikrotik seem to be ok, as mikrotik have the correc routes)


Edit:

Oh, and I forget,

don't use any vlanware bridge on the host.
2 year ago it was breaking frr-evpn.

Edit:

I don't use dhcp too! (it's not supported by proxmox anyway).
you really need static ip in /etc/network/interface, because sdn plugin is parsing them.

 

[apollo13]

This is because physical interfaces can be different on each host. (and some users also use OVS for bonding,...).

Hi, this is what I figured and I agree it makes sense. For us though it is one extra "hop" since we name all network devices consistently over the hosts so we could bind directly to eth0/bond0 (or whatever). Would you be open to relax the validation and maybe just caution users in the tooltip or so?

Thank you for your work on SDN so far.

 

[spirit]

Hi, this is what I figured and I agree it makes sense. For us though it is one extra "hop" since we name all network devices consistently over the hosts so we could bind directly to eth0/bond0 (or whatever). Would you be open to relax the validation and maybe just caution users in the tooltip or so?

Thank you for your work on SDN so far.

the current code don't handle it, but I could add an extra option for this.
The main reason was also that some users want to be able to use classic vmbrX vlan mixed with sdn, and an ethX.Y can't be plugged in 2 differents brdges.


But really that don't make any difference in term of speed.
For example, when you have firewall checkbox enable on a vm nic, you already have an extra bridge "fwbr.." created between the vmbrX && the vm tap.

If you have already used openstack, you also have this kind fof internal plumbing.

can you make a request on bugzilla.proxmox.com ? I have a lot of demand currently, and I'll forget it. (and I'm going to holiday for 1week)

 

[apollo13]

The main reason was also that some users want to be able to use classic vmbrX vlan mixed with sdn, and an ethX.Y can't be plugged in 2 differents brdges.

Oh I didn't know that ethX.Y can only be on one bridge. So the following will not work (ascii art works with monospace or viewed in the forum)?

eth0 -> eth0.2 |-> vmbr1
               |-> vmbr2

but this (like SDN does it) would?

eth0 -> vmbr0 -> vmbr0.102 |-> (zone 1) -> vlan1
                           |-> (zone 2) -> vlan2

so essentially with the bridge in between I can have a vnet in two different zones that both access the same vlan below?

can you make a request on bugzilla.proxmox.com ? I have a lot of demand currently, and I'll forget it. (and I'm going to holiday for 1week)

If what I just wrote above holds true I think that it makes little sense to support raw physical devices in SDN. Have a nice holiday!

I'd request something else on bugzilla though if you are open to it: It would be great if the "Apply" Button in the SDN view asked for confirmation (just a second layer of protection since applying results in short network hickups) and maybe a possibility to apply the changes per Server in the GUI (I know you can do it on the shell, but we are trying to limit shell access as much as possible).

 

[pille99]

Updated proxmox 7.0:
Minimum packages version (available in no-subscription-repo)


libpve-network-perl_0.6.1
ifupdown2_3.1.0-1+pmx3
pve-manager_7.0-10


-a lot of fix everywhere , so please update before reporting bugs ;)

Hi,

proxmox 7.0 include a new sdn (software delivery network) feature, it's beta for now.

I'm the main author of this feature, and I would like to have some feedback of community to improve it.

Doc is here:

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html


The main idea, to to defined virtual network at datacenter level.
The more simple example, is a vlan network. Instead of defined the vlan tag on the vm nic, we define the network at datacenter level.
This will allow to define permissions of this network. (like for a storage).

The sdn feature use a plugins, so it can be extended easily.

Currently, it's support

layer2 network
---------------------
vlan,qinq (stack vlan), vxlan

layer3 network
---------------------
vxlan+bgp-evpn , simple brige


bgp-evpn network is the most complex and true sdn network. it's need a controlller (it's using frr routing software), to manage the flow of the bridge.
It's allowing anycast routing across different vxlan network. (each proxmox host have same ip for each bgp-evpn network, and are the gateway of the vm/ct).

I think it could help too users on public servers like ovh,hetzner with different public subnet/failover ips. (you could easily diffined 1virtual network by subnet).


If users need some other sdn plugins, I'll could look to implement them in the future. (but first, I would like to have 0 bugs on current plugins)

If you have time to test it, and give some feedback on this thread, it could be wonderfull.

Thanks !


Alexandre


You can also contact me directly by email : aderumier@odiso.com



Feedback/Need to be fixed:

Gui: the vlan field on the vm/ct nic should be grey-out when a sdn vnet is choose. (keep it empty for now)

one thing i discovered, which in no document is anything written. All Videos and docus are based on 1-3 VMs but not on a Professional Network use and need. proxmox is very resources hungry,
1. for Node-to_Node communication a SDN is needed (hetzner and co, a company network with more infrastructure doesnt need)
2. the limit is round about 10 VMs per 1Gbit NIC, otherwise timeouts occure and the network is pretty much gone. i experienced it the hard way. lot of problems with performance, ping worked, websites not). after putting 10gb for sound about 25 VMs in the server, everything works again


btw: SDN = Software DEFINED Network not delivery

 

[eset]

@spirit

can you try to disable exit-node-local-routing option ? (because it's complexify the debug).

And where you found that option in my config ? I had this disabled earlier and it still is





I had docker also removed while ago. Those rules we applied as residue.

don't use any vlanware bridge on the host.

I disabled it

you really need static ip in /etc/network/interface, because sdn plugin is parsing them.

Ok moved to static also
Configs you requested

FRR

frr version 8.0.1
frr defaults datacenter
hostname pve
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_test1
 vni 4041
exit-vrf
!
router bgp 65000
 bgp router-id 10.0.1.1
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as 65000
 neighbor BGP bfd
 neighbor 10.0.1.30 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_test1
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_test1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65000 vrf vrf_test1
 bgp router-id 10.0.1.1
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty
!

Interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno0 inet manual

iface enp8s0f1 inet manual

auto enp8s0f2
iface enp8s0f2 inet manual

auto enp8s0f3
iface enp8s0f3 inet manual

iface enp8s0f0 inet manual hwaddress 00:1e:67:68:00:ce
iface enp8s0f1 inet manual hwaddress 00:1e:67:68:00:cf
iface enp8s0f2 inet manual hwaddress 00:1e:67:68:00:d0
iface enp8s0f3 inet manual hwaddress 00:1e:67:68:00:d1

auto bond0
iface bond0 inet manual
    bond-slaves enp8s0f2 enp8s0f3
    bond-miimon 100
    bond-mode 802.3ad
    mtu 8000
    bond-lacp-rate    1
#ch3 [ g5,g6 ]

auto vmbr0
iface vmbr0 inet manual
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    mtu 8000

auto vmbr0.11
iface vmbr0.11 inet static
    mtu 8000
    hwaddress 00:1e:67:00:11:01
    address 10.0.1.1/30
    gateway 10.0.1.30

auto vmbr0.110
iface vmbr0.110 inet dhcp
    mtu 8000
    hwaddress 00:1e:67:01:10:53

source /etc/network/interfaces.d/*

SDN

#version:85

auto test1
iface test1
    address 10.0.101.254/24
    hwaddress 32:1C:C7:B6:78:C5
    bridge_ports vxlan_test1
    bridge_stp off
    bridge_fd 0
    mtu 1450
    ip-forward on
    arp-accept on
    vrf vrf_test1

auto vrf_test1
iface vrf_test1
    vrf-table auto
    post-up ip route del vrf vrf_test1 unreachable default metric 4278198272

auto vrfbr_test1
iface vrfbr_test1
    bridge-ports vrfvx_test1
    bridge_stp off
    bridge_fd 0
    mtu 1450
    vrf vrf_test1

auto vrfvx_test1
iface vrfvx_test1
    vxlan-id 4041
    vxlan-local-tunnelip 10.0.1.1
    bridge-learning off
    bridge-arp-nd-suppress on
    mtu 1450

auto vxlan_test1
iface vxlan_test1
    vxlan-id 4090
    vxlan-local-tunnelip 10.0.1.1
    bridge-learning off
    bridge-arp-nd-suppress on
    mtu 1450

IP ROUTE TABLE ALL

10.0.101.0/24 dev test1 table vrf_test1 proto kernel scope link src 10.0.101.254
local 10.0.101.254 dev test1 table vrf_test1 proto kernel scope host src 10.0.101.254
broadcast 10.0.101.255 dev test1 table vrf_test1 proto kernel scope link src 10.0.101.254
default via 10.0.1.30 dev vmbr0.11 proto kernel onlink
10.0.1.0/30 dev vmbr0.11 proto kernel scope link src 10.0.1.1
10.0.10.0/24 dev vmbr0.110 proto kernel scope link src 10.0.10.252
10.0.101.0/24 nhid 14 dev test1 proto bgp metric 20
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
local 10.0.1.1 dev vmbr0.11 table local proto kernel scope host src 10.0.1.1
broadcast 10.0.1.3 dev vmbr0.11 table local proto kernel scope link src 10.0.1.1
local 10.0.10.252 dev vmbr0.110 table local proto kernel scope host src 10.0.10.252
broadcast 10.0.10.255 dev vmbr0.110 table local proto kernel scope link src 10.0.10.252
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.17.0.1 dev docker0 table local proto kernel scope host src 172.17.0.1
broadcast 172.17.255.255 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown
anycast fe80:: dev vrfbr_test1 table vrf_test1 proto kernel metric 0 pref medium
anycast fe80:: dev test1 table vrf_test1 proto kernel metric 0 pref medium
local fe80::301c:c7ff:feb6:78c5 dev test1 table vrf_test1 proto kernel metric 0 pref medium
local fe80::54d8:13ff:feed:6d1c dev vrfbr_test1 table vrf_test1 proto kernel metric 0 pref medium
fe80::/64 dev vrfbr_test1 table vrf_test1 proto kernel metric 256 pref medium
fe80::/64 dev test1 table vrf_test1 proto kernel metric 256 pref medium
multicast ff00::/8 dev vrfbr_test1 table vrf_test1 proto kernel metric 256 pref medium
multicast ff00::/8 dev test1 table vrf_test1 proto kernel metric 256 pref medium
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0.11 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0.110 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fe80:: dev vmbr0 table local proto kernel metric 0 pref medium
local fe80::21e:67ff:fe00:1101 dev vmbr0.11 table local proto kernel metric 0 pref medium
local fe80::21e:67ff:fe01:1053 dev vmbr0.110 table local proto kernel metric 0 pref medium
local fe80::ecd8:9fff:fe43:5a3b dev vmbr0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev vmbr0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev vmbr0.11 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev vmbr0.110 table local proto kernel metric 256 pref medium

sh ip route

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 10.0.1.0/30 is directly connected, vmbr0.11, 00:05:33
C>* 10.0.10.0/24 is directly connected, vmbr0.110, 00:05:24
B>* 10.0.101.0/24 [20/0] is directly connected, test1 (vrf vrf_test1), weight 1, 00:05:23

sh bgp l2vpn evpn

BGP table version is 4, local router ID is 10.0.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.1.1:2
*> [5]:[0]:[0]:[0.0.0.0]
                    10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4041 Rmac:56:d8:13:ed:6d:1c
*> [5]:[0]:[0]:[::] 10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4041 Rmac:56:d8:13:ed:6d:1c
Route Distinguisher: 10.0.1.1:3
*> [2]:[0]:[48]:[de:1f:16:7e:1a:ec]
                    10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4090
*> [2]:[0]:[48]:[de:1f:16:7e:1a:ec]:[32]:[10.0.101.1]
                    10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4090 RT:65000:4041 Rmac:56:d8:13:ed:6d:1c
*> [2]:[0]:[48]:[de:1f:16:7e:1a:ec]:[128]:[fe80::dc1f:16ff:fe7e:1aec]
                    10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4090
*> [3]:[0]:[32]:[10.0.1.1]
                    10.0.1.1(pve)                      32768 i
                    ET:8 RT:65000:4090

Displayed 6 out of 6 total prefixes

Server rebooted. Still nothing. Packets stops on 10.0.1.1. There is issue between BGP on default (10.0.1.1) and the vrf in 10.0.101.0/24

I also tested VLAN. Set on vmbr0 vlan aware and created additional interface with VLAN tag. Added IP on proxmox newly created interface for that vlan which is the gateway of subnet that belongs to the VNet in the specific Zone of type VLAN. VM can ping 10.0.103.254 (subnet gateway and IP of the Proxmox interface). But MikroTik can't ping the VM. It can only reach 10.103.254 and nothing behind that. Still trace route stops on 10.0.1.1 .. can't say why.
And that doesn't make sense when vmbr0 is a bridge to which vmbr0.11 belongs with newly created vmbr0.1103 so the both are in the sam bridge

 

[apollo13]

I'd request something else on bugzilla though if you are open to it: It would be great if the "Apply" Button in the SDN view asked for confirmation (just a second layer of protection since applying results in short network hickups) and maybe a possibility to apply the changes per Server in the GUI (I know you can do it on the shell, but we are trying to limit shell access as much as possible).

I thought about this a bit more. If "Apply" behaved like "Bulk Migrate" in the sense that it presented a list of nodes to apply on, we'd "implicitly" get confirmation and also get the "per host apply" in one fix

 

[spirit]

I thought about this a bit more. If "Apply" behaved like "Bulk Migrate" in the sense that it presented a list of nodes to apply on, we'd "implicitly" get confirmation and also get the "per host apply" in one fix

if you use use "network apply" on host network panel, it'll regenerate also the sdn configuration for this host only.

then sdn apply, basically, call each host network reload, once by once.

 

[apollo13]

if you use use "network apply" on host network panel, it'll regenerate also the sdn configuration for this host only.

Ah right, so I guess all I am really after is a confirmation dialog for "Apply" since it is a really disruptive action Thanks for your work on SDN so far, it really works great.

 

[spirit]

Ah right, so I guess all I am really after is a confirmation dialog for "Apply" since it is a really disruptive action Thanks for your work on SDN so far, it really works great.

Will try to implemend this. (BTW, it shouldn't be disruptive, reloading of conf don't break currently running network connnections)

 

[apollo13]

Will try to implemend this. (BTW, it shouldn't be disruptive, reloading of conf don't break currently running network connnections)

Interesting, we certainly saw issues when reloading. Might have been due how our Ceph storage is setup though.

 

[eset]

@spirit so my case is a dead one?

 

[spirit]

@spirit so my case is a dead one?

Hi,sorry, I dind't see your last reply and I was super busy last months.

looking at your last reponse:

auto vmbr0.11
iface vmbr0.11 inet static
    mtu 8000
    hwaddress 00:1e:67:00:11:01
    address 10.0.1.1/30
    gateway 10.0.1.30

you shouldn't use tag on vmbrX, if the vmbr is not vlan aware . (and vlan-aware is not well compatible with evpn currently)

just try to tag the bond directly

auto bond0.11
iface bond0.11 inet static
    mtu 8000
    address 10.0.1.1/30
    gateway 10.0.1.30

Another thing to check, do you have enabled ipv4 forwarding ?

sysctl -w net.ipv4.ip_forward=1
?


for testing, try to do a ping from a vm to external world,
then do a tcpdump in the vrf

"tcpdump -i vrf_test1 icmp"

and check if the packet is routed to the default vrf with

"tcpdump -i bond0.11 icmp"

if you see the packets on both side, it's ok.

 

[eset]

Hi,sorry, I dind't see your last reply and I was super busy last months.

looking at your last reponse:

auto vmbr0.11
iface vmbr0.11 inet static
    mtu 8000
    hwaddress 00:1e:67:00:11:01
    address 10.0.1.1/30
    gateway 10.0.1.30

you shouldn't use tag on vmbrX, if the vmbr is not vlan aware . (and vlan-aware is not well compatible with evpn currently)

just try to tag the bond directly

auto bond0.11
iface bond0.11 inet static
    mtu 8000
    address 10.0.1.1/30
    gateway 10.0.1.30

Another thing to check, do you have enabled ipv4 forwarding ?

sysctl -w net.ipv4.ip_forward=1
?


for testing, try to do a ping from a vm to external world,
then do a tcpdump in the vrf

"tcpdump -i vrf_test1 icmp"

and check if the packet is routed to the default vrf with

"tcpdump -i bond0.11 icmp"

if you see the packets on both side, it's ok.

Finally. It works. Thank you a lot :/ so long have been waiting :/

 

[spirit]

Finally. It works. Thank you a lot :/ so long have been waiting :/

so, what was the problem ? (I'll add a note in the documentation)

 

[eset]

so, what was the problem ? (I'll add a note in the documentation)

after removing the vmbr0.11 and moved vlan to bond0.11 and reload `networking` service I started to work.
Btw is there a way to "smuggle" a DNS server to that SDN? I Have a PowerDNS on Proxmox host working. It works for LAN users. But how I can query from VM in SDN that DNS. Ping works but there is no port forwarding So Im' thinking how to manage that.

 

[spirit]

after removing the vmbr0.11 and moved vlan to bond0.11 and reload `networking` service I started to work.
Btw is there a way to "smuggle" a DNS server to that SDN? I Have a PowerDNS on Proxmox host working. It works for LAN users. But how I can query from VM in SDN that DNS. Ping works but there is no port forwarding So Im' thinking how to manage that.

I'm currently working on this at work for the coming months, I think the best way could be to deploy a dns resolver on each host, and use the anycast gateway ip as dns in the vm. It should work, maybe with some tuning in systemd service to listen inside the vrf. "execstart = ip exec vrf myzone /usr/bin/dnsbinary" .


But if you want to reach external network from evpn, you need to enable exit-nodes.

 

[eset]

I'm currently working on this at work for the coming months, I think the best way could be to deploy a dns resolver on each host, and use the anycast gateway ip as dns in the vm. It should work, maybe with some tuning in systemd service to listen inside the vrf. "execstart = ip exec vrf myzone /usr/bin/dnsbinary" .

Sounds like a bit workaround than a complex solution. In GCP Cloud they use loopback IP 169.254.169.254 for that which probably move packets to a DNS service in their network.

But if you want to reach external network from evpn, you need to enable exit-nodes.

You mean this ?



Exit Nodes local routing box?