EVPN in 8.1 requires MTU tweak for guest communication

[pratclot]

Hello everyone!

I am trying to setup a PVE cluster at home with 2 machines. A couple months ago I managed to get a basic setup going with 8.0-2, and now trying with 8.1-1, but there seems to be a new problem with networking that I did not encounter before.

After the cluster is created, I add EVPN network to it. If I understand correctly, this solution will allow my VM's (I used this image) to talk to each other even if they are running on separate nodes. Plus, I add a BGP controller that allows me to automatically have a route to these VM's on my laptop. After this I verified that I can ssh to VM's from laptop, plus the VM's can ping each other while they are scheduled on different nodes (but did not try anything else).

I do not know if it is important, but I set up static IP addresses on the VM's via Cloud-Init.

Then I tried to install some packages on the VM's, and they tell me that DNS is not working, like that:

# host google.com
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.4.4#53: timed out
;; no servers could be reached

I launch tcpdump on the node with the VM, and verify that these requests have a response (I called the zone "evpn1"):

# tcpdump -i any src host dns.google and dst host 10.15.0.30 and port 53
21:46:53.065892 xvrf_evpn1 Out IP dns.google.domain > 10.15.0.30.36548: 59133 1/0/0 A 142.250.186.174 (44)
21:46:53.065895 xvrfp_evpn1 In  IP dns.google.domain > 10.15.0.30.36548: 59133 1/0/0 A 142.250.186.174 (44)

I do not know much about VRF, so not sure what to check next. I also added a "Simple" zone to SDN and attached the VM to this network, and DNS and everything else started to work through this network. Because of that, I kept the configuration with 2 networks.

Then I needed my VM's to talk to each other over EVPN (an HTTP request), and again, they could not (curl just hangs for some time):

# curl https://10.15.0.30:6443 -v
*   Trying 10.15.0.30:6443...
* Connected to 10.15.0.30 (10.15.0.30) port 6443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* Recv failure: Connection reset by peer
* OpenSSL SSL_connect: Connection reset by peer in connection to 10.15.0.30:6443 
* Closing connection 0
curl: (35) Recv failure: Connection reset by peer

I tried to lower MTU on the network adapters from default 1500 to 1450 (this is what VXLAN has, but this change does not make a lot of sense I think) and the VM's were able to communicate to each other. I then disconnected the "Simple" network to check if communication with outer world would work ("host google.com"), but it was not the case.

I also verified that the VM's can talk to each other with MTU 1500 when they run on the same node. This is probably a useless check though.

I do not remember this issue on 8.0-2, but of course maybe I did something differently this time (but I cannot figure out what, haha). Sorry for the long post!

 

[spirit]

evpn is a vxlan network, so yes, it need mtu 1450 if your physical network is 1500.
it's was always in the documentation since day one.

vxlan need 50 bytes for encapsulation.

If not defined the the zone, the vnets are correctly created with mtu 1450 by default.

But, the vms interfaces still need to be configured to 1450 too.
in your vms nic configuration, if you choose mtu:1 (inherit mtu from bridge) and you use virtio-nic, they should be configured correctly to 1450 in the guest os.

 

[pratclot]

Hey spirit, thanks a lot for your answer, I really liked the "MTU 1" shortcut!

There are a couple of things I do not understand:
- even with lower MTU the VM's still cannot receive DNS responses (any responses really, apart from ICMP), while tcpdump confirms they arrive to xvrfp_XXXX interface. Is it possible at all to reach Internet via EVPN? (I am pretty sure the answer is "yes", but then I do not understand why the node cannot deliver packets to the VM's).
- right after connecting VM's to EVPN network, without MTU tweaks on the network adapters, I was able to connect to them via ssh from my laptop (proxmox nodes and the laptop are connected to the same Wi-Fi which provides Internet access). I am happy that it is this way, however I expect the traffic should behave similarly to DNS reponses, i.e. not reaching the VM's. But it works, and this is what I recall happened with 8.0-2, I did not even know about MTU requirement.

 

[spirit]

- even with lower MTU the VM's still cannot receive DNS responses (any responses really, apart from ICMP), while tcpdump confirms they arrive to xvrfp_XXXX interface.

"xvrfp_XXXX" is a special interface, when you enable option "enable local-routing" in the evpn zone. It's only needed if you want to reach vm ip from the host management ip.
Do you really need this ? (Until you proxmox management ip is the dns server, you shouldn't need it)

Is it possible at all to reach Internet via EVPN? (I am pretty sure the answer is "yes", but then I do not understand why the node cannot deliver packets to the VM's).

yes, sure. you need to define an exit-node. (maybe with snat if you use private ip in evpn)
The exit-node is routing between evpn network && real network (and for public, forwarding to outside through his own gateway )

- right after connecting VM's to EVPN network, without MTU tweaks on the network adapters, I was able to connect to them via ssh from my laptop (proxmox nodes and the laptop are connected to the same Wi-Fi which provides Internet access). I am happy that it is this way, however I expect the traffic should behave similarly to DNS reponses, i.e. not reaching the VM's. But it works, and this is what I recall happened with 8.0-2, I did not even know about MTU requirement.

If you don't have an exit-node , this is strange that it's working.


an even, with an exit-node, you need to be able to reach to evpn subnet from your wifi network, so in your router or locally on your laptop,

you should have a static route like:

ip route add <evpnsubnet> gw <exit-no-realip>



Where is your dns server ? on the wifi network ?

Can you share your /etc/pve/sdn/*.cfg && /etc/network/interfaces ?

 

[pratclot]

Hey spirit,

I am happy to report that disabling "enable local-routing" option made VM's receive traffic from Internet over EVPN! I mistakenly thought this may be needed to access EVPN from "real" network, even though it is explained in the docs.

Thanks a lot for you assistance and detailed explanation of the features. It is my first post here, but I feel lucky that there are people like yourself here who can provide expert advice so quickly (and most importantly for free, haha).

 

[spirit]

Hey spirit,

I am happy to report that disabling "enable local-routing" option made VM's receive traffic from Internet over EVPN! I mistakenly thought this may be needed to access EVPN from "real" network, even though it is explained in the docs.

Thanks a lot for you assistance and detailed explanation of the features. It is my first post here, but I feel lucky that there are people like yourself here who can provide expert advice so quickly (and most importantly for free, haha).

you're welcome

(I really need to improve the official doc with somes examples ^_^ )