Proxmox SDN EVPN SNAT mangling src port on return traffic after TCP connection established

D

darragh

New Member
Aug 25, 2024
6
0
1
I spent 2 days trying to debug this but at my wits end. Any help would be greatly appreciated!

It's my understanding that Proxmox SDN EVPN supports SNAT out of the box because it's a layer 3 SDN, provided you specify a single exit node or a primary exit node, and you select the SNAT option in the subnet configuration. And this does work for pings and establishing TCP connections "nc -cw 1 xxx.xxx.xxx.xxx". But there's some connection tracking behaviour I do not understand that's mangling the port of the remote http server mid-way through the network stack.

It's entirely possible I'm missing something obvious as I don't meaningfully understand the SDN stack, BGP and FRR


Code: 
root@unai-prod-pve-1-hetzner-fsn1-ax42-2421882 ~ # tcpdump -nni any ip and not port 51820  and not port 5405 and not port 8006 and not port 4784 and not port 179 and not port 8080 and not port 85 and not icmp6 and not arp and not port 5900 and not port 4789 and not port 22 and not port 53
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:39:33.572485 tap101i0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.572488 fwln101i0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.572490 fwpr101p0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.572491 vnet1 In  IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.572509 vmbr0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.572512 enp6s0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.589748 enp6s0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589753 vmbr0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589777 xvrf_evpn1 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589790 xvrfp_evpn1 In  IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589800 vnet1 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589804 fwpr101p0 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589806 fwln101i0 P   IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589809 tap101i0 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.589986 tap101i0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.589993 fwln101i0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.589999 fwpr101p0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.590002 vnet1 In  IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.590022 vmbr0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.590025 enp6s0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.590047 tap101i0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.590048 fwln101i0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.590049 fwpr101p0 P   IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.590050 vnet1 In  IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.590054 vmbr0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.590055 enp6s0 Out IP 144.76.xxx.xxx.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.607474 enp6s0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [.], ack 78, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607479 vmbr0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [.], ack 78, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607500 xvrf_evpn1 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [.], ack 78, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607513 xvrfp_evpn1 In  IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [.], ack 78, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607529 vnet1 Out IP 216.58.212.238.473 > 10.0.0.66.50088: Flags [.], ack 984070366, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607533 fwpr101p0 Out IP 216.58.212.238.473 > 10.0.0.66.50088: Flags [.], ack 1, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607534 fwln101i0 P   IP 216.58.212.238.473 > 10.0.0.66.50088: Flags [.], ack 1, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607538 tap101i0 Out IP 216.58.212.238.473 > 10.0.0.66.50088: Flags [.], ack 1, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.607725 tap101i0 P   IP 10.0.0.66.50088 > 216.58.212.238.473: Flags [R], seq 984070366, win 0, length 0
02:39:33.607732 fwln101i0 Out IP 10.0.0.66.50088 > 216.58.212.238.473: Flags [R], seq 984070366, win 0, length 0
02:39:33.607735 fwpr101p0 P   IP 10.0.0.66.50088 > 216.58.212.238.473: Flags [R], seq 984070366, win 0, length 0
02:39:33.607738 vnet1 In  IP 10.0.0.66.50088 > 216.58.212.238.473: Flags [R], seq 984070366, win 0, length 0
02:39:33.607759 vmbr0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [R], seq 984070366, win 0, length 0
02:39:33.607764 enp6s0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [R], seq 984070366, win 0, length 0
02:39:33.616112 enp6s0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [P.], seq 1:774, ack 78, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773: HTTP: HTTP/1.1 301 Moved Permanently
02:39:33.616119 vmbr0 In  IP 216.58.212.238.80 > 144.76.xxx.xxx.50088: Flags [P.], seq 1:774, ack 78, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773: HTTP: HTTP/1.1 301 Moved Permanently
02:39:33.616152 xvrf_evpn1 Out IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [P.], seq 1:774, ack 78, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773: HTTP: HTTP/1.1 301 Moved Permanently
02:39:33.616166 xvrfp_evpn1 In  IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [P.], seq 1:774, ack 78, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773: HTTP: HTTP/1.1 301 Moved Permanently
02:39:33.616188 vnet1 Out IP 216.58.212.238.509 > 10.0.0.66.50088: Flags [P.], seq 889927949:889928722, ack 984070366, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773
02:39:33.616194 fwpr101p0 Out IP 216.58.212.238.509 > 10.0.0.66.50088: Flags [P.], seq 0:773, ack 1, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773
02:39:33.616196 fwln101i0 P   IP 216.58.212.238.509 > 10.0.0.66.50088: Flags [P.], seq 0:773, ack 1, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773
02:39:33.616202 tap101i0 Out IP 216.58.212.238.509 > 10.0.0.66.50088: Flags [P.], seq 0:773, ack 1, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773
02:39:33.616489 tap101i0 P   IP 10.0.0.66.50088 > 216.58.212.238.509: Flags [R], seq 984070366, win 0, length 0
02:39:33.616498 fwln101i0 Out IP 10.0.0.66.50088 > 216.58.212.238.509: Flags [R], seq 984070366, win 0, length 0
02:39:33.616503 fwpr101p0 P   IP 10.0.0.66.50088 > 216.58.212.238.509: Flags [R], seq 984070366, win 0, length 0
02:39:33.616507 vnet1 In  IP 10.0.0.66.50088 > 216.58.212.238.509: Flags [R], seq 984070366, win 0, length 0
02:39:33.616538 vmbr0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [R], seq 984070366, win 0, length 0
02:39:33.616544 enp6s0 Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [R], seq 984070366, win 0, length 0


Code: 
root@vm1:~# tcpdump -nni any not port 22 and not arp and not port 53
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:39:33.199701 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [S], seq 984070288, win 65170, options [mss 1330,sackOK,TS val 109211545 ecr 0,nop,wscale 7], length 0
02:39:33.217152 eth0  In  IP 216.58.212.238.80 > 10.0.0.66.50088: Flags [S.], seq 889927948, ack 984070289, win 65535, options [mss 1412,sackOK,TS val 4190072347 ecr 109211545,nop,wscale 8], length 0
02:39:33.217185 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [.], ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 0
02:39:33.217258 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211562 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.234900 eth0  In  IP 216.58.212.238.473 > 10.0.0.66.50088: Flags [.], ack 984070366, win 1050, options [nop,nop,TS val 4190072365 ecr 109211562], length 0
02:39:33.234921 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.473: Flags [R], seq 984070366, win 0, length 0
02:39:33.243608 eth0  In  IP 216.58.212.238.509 > 10.0.0.66.50088: Flags [P.], seq 889927949:889928722, ack 984070366, win 1050, options [nop,nop,TS val 4190072373 ecr 109211562], length 773
02:39:33.243671 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.509: Flags [R], seq 984070366, win 0, length 0
02:39:33.438441 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109211783 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1
02:39:33.455808 eth0  In  IP 216.58.212.238.335 > 10.0.0.66.50088: Flags [.], ack 984070366, win 1050, options [nop,nop,TS val 4190072586 ecr 109211783,nop,nop,sack 1 {4294967220:1}], length 0
02:39:33.455867 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.335: Flags [R], seq 984070366, win 0, length 0
02:39:33.462141 eth0  In  IP 216.58.212.238.50 > 10.0.0.66.50088: Flags [P.], seq 889927949:889928722, ack 984070366, win 1050, options [nop,nop,TS val 4190072592 ecr 109211783], length 773
02:39:33.462173 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.50: Flags [R], seq 984070366, win 0, length 0
02:39:33.613376 eth0  In  IP 216.58.212.238.291 > 10.0.0.66.53020: Flags [P.], seq 265866507:265867280, ack 3520657297, win 1050, options [nop,nop,TS val 1584285365 ecr 109198055], length 773
02:39:33.613408 eth0  Out IP 10.0.0.66.53020 > 216.58.212.238.291: Flags [R], seq 3520657297, win 0, length 0
02:39:33.662129 eth0  Out IP 10.0.0.66.50088 > 216.58.212.238.80: Flags [P.], seq 1:78, ack 1, win 510, options [nop,nop,TS val 109212007 ecr 4190072347], length 77: HTTP: GET / HTTP/1.1


From a different attempt but same behaviour:
Code: 
[1750475024.871120]        [NEW] tcp      6 120 SYN_SENT src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=80 [UNREPLIED] src=216.58.212.238 dst=144.76.xxx.xxx sport=80 dport=48208
[1750475024.887953]     [UPDATE] tcp      6 60 SYN_RECV src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=80 src=216.58.212.238 dst=144.76.xxx.xxx sport=80 dport=48208
[1750475024.888067]     [UPDATE] tcp      6 86400 ESTABLISHED src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=80 src=216.58.212.238 dst=144.76.xxx.xxx sport=80 dport=48208 [ASSURED]
[1750475024.905199]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=411
[1750475024.905479]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=411 delta-time=0
[1750475024.913022]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=241
[1750475024.913274]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=241 delta-time=0
[1750475025.131048]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=229
[1750475025.131241]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=229 delta-time=0
[1750475025.133071]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=30
[1750475025.133268]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=30 delta-time=0
[1750475025.355366]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=124
[1750475025.355643]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=124 delta-time=0
[1750475025.357135]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=333
[1750475025.357398]    [DESTROY] tcp      6 300 CLOSE src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=333 delta-time=0
[1750475025.795341]        [NEW] tcp      6 300 ESTABLISHED src=216.58.212.238 dst=10.0.0.66 sport=80 dport=48208 [UNREPLIED] src=10.0.0.66 dst=216.58.212.238 sport=48208 dport=200


[ATTACH type="full" width="500px" size="1158x1462"]87350[/ATTACH][ATTACH type="full" width="500px" size="880x662"]87351[/ATTACH][ATTACH type="full" width="500px" size="900x1492"]87352[/ATTACH]
 

Attachments

  • evpn controller config.png
    evpn controller config.png
    116.1 KB · Views: 1
  • evpn subnet.png
    evpn subnet.png
    68.8 KB · Views: 1
  • evpn config.png
    evpn config.png
    170.6 KB · Views: 1
Last edited:
For some additional context: I have disabled the Proxmox firewall at datacenter, host and VM, and vnet levels. I'm also running the SDN on top of 192.168.0.0/16 which is configured with Wireguard, which is why I choose the 1370 MTU.

Do you have any suggestions for how to investigate this issue or what next steps I should take? I know I can create a Simple SDN to grant network activity to my VMs but I would really like to understand what I'm doing wrong here.
 
Setting the following seems to correct the behaviour when the firewall is disabled (but turning the firewall on at the datacenter level while disabling it everywhere else still results in broken SNAT routing):
sysctl -w net.netfilter.nf_conntrack_tcp_loose=0

But why would the port change in the middle of the networking stack? I haven't configured any iptables translations or anything that might explain the behaviour
 
Last edited:
You must log in or register to reply here.
Top Bottom