Hello everybody,
I'm currently planning the deployment of a PVE cluster, span over multiple datacenters so I'm testing a SDN configuration that allows to define a dedicated subnet for the VMs, conceptually like a virtual datacenter for VMs, with full L3 routing with the real DCs.
Each datacenter has static routes to route the 192.168.64.0/24 network into the nearest node of the PVE cluster, and now:
This is the network configuration on PVE01 node for EVPN model:
This is the current configuration on PVE03 node:
IMHO the strangest part is that in EVPN model I never see any traffic flowing in UDP/4789 tunnel between PVE nodes...
I thought that EVPN mode should be based on the same UDP/4789 tunnel as VXLAN mode for intra-cluster communications, isn't it?
Am I missing something?
Thanks in advance.
I'm currently planning the deployment of a PVE cluster, span over multiple datacenters so I'm testing a SDN configuration that allows to define a dedicated subnet for the VMs, conceptually like a virtual datacenter for VMs, with full L3 routing with the real DCs.
- I'm using PVE 8.2.7
- Each datacenter is already connected with the others through IPSec tunnels and static routing.
- Each datacenter has a node of the PVE cluster
- The VMs hosted by each node of the PVE cluster:
- must be connected to the same dedicated SDN
- must be able to reach the VMs hosted by the other PVE nodes of the same cluster
- must be able to reach the real servers in both datacenters (without using SNAT)
- must be reachable by real servers from both datacenters
- the communication between VMs hosted by different PVE nodes is OK, as expected
- I can see traffic flowing in UDP/4789 tunnel between PVE nodes, as expected
- the communication from VMs to servers in real DCs is not working, as expected
- the communication from servers in real DCs to VMs is not working, as expected
- the VXLAN network is shared between remote PVE cluster nodes, but isolated from networks in datacenters
- ICMP is NOT working between VMs and their GW (192.168.64.1) (is it expected ?)
Each datacenter has static routes to route the 192.168.64.0/24 network into the nearest node of the PVE cluster, and now:
- the communication between VMs hosted by different PVE nodes is NOT working
- I CANNOT see traffic flowing in UDP/4789 tunnel between PVE nodes
- the communication from VMs to servers in real DCs is working only on same site
- from VMs hosted by PVE01 to DC1
- from VMs hosted by PVE03 to DC3
- the communication from servers in real DCs to VMs is working only on same site
- from DC1 to VMs hosted by PVE01
- from DC3 to VMs hosted by PVE03
- ICMP is working between VMs and their GW (192.168.64.1)
Code:
root@pve01:~# cat /etc/pve/sdn/*
evpn: evpnctrl
asn 65000
peers 192.168.127.78,192.168.230.12
subnet: evpnzone-192.168.64.0-24
vnet evpnnet
gateway 192.168.64.1
vnet: evpnnet
zone evpnzone
alias EVPN Network
tag 11000
evpn: evpnzone
controller evpnctrl
vrf-vxlan 10000
exitnodes pve03,pve01
ipam pve
mac BC:24:11:F9:5F:A3
root@pve01:~#
This is the network configuration on PVE01 node for EVPN model:
Code:
root@pve01:~# cat /etc/network/interfaces.d/sdn
#version:47
auto evpnnet
iface evpnnet
address 192.168.64.1/24
hwaddress BC:24:11:F9:5F:A3
bridge_ports vxlan_evpnnet
bridge_stp off
bridge_fd 0
mtu 1450
alias EVPN Network
ip-forward on
arp-accept on
vrf vrf_evpnzone
auto vrf_evpnzone
iface vrf_evpnzone
vrf-table auto
post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272
auto vrfbr_evpnzone
iface vrfbr_evpnzone
bridge-ports vrfvx_evpnzone
bridge_stp off
bridge_fd 0
mtu 1450
vrf vrf_evpnzone
auto vrfvx_evpnzone
iface vrfvx_evpnzone
vxlan-id 10000
vxlan-local-tunnelip 192.168.230.12
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
auto vxlan_evpnnet
iface vxlan_evpnnet
vxlan-id 11000
vxlan-local-tunnelip 192.168.230.12
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
root@pve01:~#
root@pve01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0np0
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e4:43:4b:61:f0:b1 brd ff:ff:ff:ff:ff:ff
altname enp1s0f1np1
4: enp132s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether f4:c7:aa:43:1b:08 brd ff:ff:ff:ff:ff:ff
5: eno3np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e4:43:4b:61:f0:b2 brd ff:ff:ff:ff:ff:ff
altname enp1s0f2np2
6: enp132s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether f4:c7:aa:43:1b:09 brd ff:ff:ff:ff:ff:ff
7: eno4np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e4:43:4b:61:f0:b3 brd ff:ff:ff:ff:ff:ff
altname enp1s0f3np3
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.12/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::e643:4bff:fe61:f0b0/64 scope link
valid_lft forever preferred_lft forever
10: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.1/24 scope global evpnnet
valid_lft forever preferred_lft forever
inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
valid_lft forever preferred_lft forever
11: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
link/ether 3a:c2:31:9d:7b:25 brd ff:ff:ff:ff:ff:ff
13: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a04c:afff:fec0:4e12/64 scope link
valid_lft forever preferred_lft forever
14: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
link/ether 2a:af:f4:50:3e:2b brd ff:ff:ff:ff:ff:ff
25: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
link/ether 16:6e:62:87:b7:4f brd ff:ff:ff:ff:ff:ff
26: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
root@pve01:~#
root@pve01:~# ip r
default via 192.168.230.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 47 dev evpnnet proto bgp metric 20
192.168.230.0/24 dev vmbr0 proto kernel scope link src 192.168.230.12
root@pve01:~#
This is the current configuration on PVE03 node:
Code:
root@pve03:~# cat /etc/network/interfaces.d/sdn
#version:47
auto evpnnet
iface evpnnet
address 192.168.64.1/24
hwaddress BC:24:11:F9:5F:A3
bridge_ports vxlan_evpnnet
bridge_stp off
bridge_fd 0
mtu 1450
alias EVPN Network
ip-forward on
arp-accept on
vrf vrf_evpnzone
auto vrf_evpnzone
iface vrf_evpnzone
vrf-table auto
post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272
auto vrfbr_evpnzone
iface vrfbr_evpnzone
bridge-ports vrfvx_evpnzone
bridge_stp off
bridge_fd 0
mtu 1450
vrf vrf_evpnzone
auto vrfvx_evpnzone
iface vrfvx_evpnzone
vxlan-id 10000
vxlan-local-tunnelip 192.168.127.78
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
auto vxlan_evpnnet
iface vxlan_evpnnet
vxlan-id 11000
vxlan-local-tunnelip 192.168.127.78
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
root@pve03:~#
root@pve03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 7c:83:34:be:1c:ac brd ff:ff:ff:ff:ff:ff
4: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b0:dc:ef:b8:14:e7 brd ff:ff:ff:ff:ff:ff
altname wlp0s20f3
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
inet 192.168.127.78/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::7e83:34ff:febe:1cab/64 scope link
valid_lft forever preferred_lft forever
7: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.1/24 scope global evpnnet
valid_lft forever preferred_lft forever
inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
valid_lft forever preferred_lft forever
8: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
link/ether 1e:81:16:88:8e:36 brd ff:ff:ff:ff:ff:ff
10: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4cc:acff:fee5:52db/64 scope link
valid_lft forever preferred_lft forever
11: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
link/ether c6:27:b2:20:8d:00 brd ff:ff:ff:ff:ff:ff
20: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
link/ether 8a:a2:62:d2:b8:3d brd ff:ff:ff:ff:ff:ff
21: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
root@pve03:~#
root@pve03:~# ip r
default via 192.168.127.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 31 dev evpnnet proto bgp metric 20
192.168.127.0/24 dev vmbr0 proto kernel scope link src 192.168.127.78
root@pve03:~#
IMHO the strangest part is that in EVPN model I never see any traffic flowing in UDP/4789 tunnel between PVE nodes...
I thought that EVPN mode should be based on the same UDP/4789 tunnel as VXLAN mode for intra-cluster communications, isn't it?
Am I missing something?
Thanks in advance.
Last edited: