Bridge network setup -> very low MTU from VMs.. why?

Feb 23, 2022
9
1
8
28
Hi,

Running a proxmox setup where all VMs are bridged to a single physical interface in host, all with default MTU of 1500.

We noticed that websocket connections are closing abruptly, and running the following bash script:

Bash:
size=1272
while ping -s $size -c1 -M do google.com >&/dev/null; do
  ((size+=4))
done
echo "Max MTU size: $((size-4+28))"

Showed an MTU of 1296 in VMs, while running in host shows 1500.

Any ideas why? Have limited understanding in MTU/MSS, but 1296 is quite low..

ifconfig on host shows everything on 1500:
Code:
enp152s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 3c:ec:ef:b7:fa:8c  txqueuelen 40000  (Ethernet)
        RX packets 13086020619  bytes 10072373360184 (9.1 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16793858705  bytes 21713322348584 (19.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fwbr100i0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 52:fa:5f:9f:6f:e1  txqueuelen 1000  (Ethernet)
        RX packets 99617  bytes 7196620 (6.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fwln100i0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 52:fa:5f:9f:6f:e1  txqueuelen 1000  (Ethernet)
        RX packets 6574036135  bytes 4458682824606 (4.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4943211571  bytes 17199642687289 (15.6 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fwpr100p0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether a2:29:04:b5:16:07  txqueuelen 1000  (Ethernet)
        RX packets 4943211570  bytes 17199642687223 (15.6 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6574036133  bytes 4458682820130 (4.0 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 572662  bytes 70686791 (67.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 572662  bytes 70686791 (67.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tailscale0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1280
        inet 100.93.48.90  netmask 255.255.255.255  destination 100.93.48.90
        inet6 fd7a:115c:a1e0::501:305a  prefixlen 128  scopeid 0x0<global>
        inet6 fe80::d98d:6ec5:2612:593b  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 656845  bytes 110539025 (105.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 543198  bytes 426247653 (406.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap100i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 46:45:f4:6f:6f:df  txqueuelen 1000  (Ethernet)
        RX packets 4943216384  bytes 17199643015080 (15.6 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6574078516  bytes 4458684523783 (4.0 TiB)
        TX errors 0  dropped 890 overruns 0  carrier 0  collisions 0

tap101i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether ce:38:e0:2c:7f:53  txqueuelen 1000  (Ethernet)
        RX packets 603091249  bytes 454414676344 (423.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1003225136  bytes 1944134992270 (1.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap102i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 6e:f9:c3:7a:7e:21  txqueuelen 1000  (Ethernet)
        RX packets 72500042  bytes 122667409552 (114.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 108024949  bytes 124454557216 (115.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap103i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 4e:c8:6e:2c:5d:17  txqueuelen 1000  (Ethernet)
        RX packets 58501  bytes 4549026 (4.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 80886  bytes 364295030 (347.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 1.1.1.1  netmask 255.255.255.248  broadcast 0.0.0.0
        inet6 fe80::3eec:efff:feb7:fa8c  prefixlen 64  scopeid 0x20<link>
        ether 3c:ec:ef:b7:fa:8c  txqueuelen 1000  (Ethernet)
        RX packets 6126418  bytes 9702077777 (9.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4866497  bytes 9877701568 (9.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

same thing with VM:
Code:
enp6s18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 103.107.198.45  netmask 255.255.255.248  broadcast 103.107.198.47
        inet6 fe80::be24:11ff:fe38:5f5c  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:38:5f:5c  txqueuelen 1000  (Ethernet)
        RX packets 110679489  bytes 127374723957 (127.3 GB)
        RX errors 0  dropped 1155  overruns 0  frame 0
        TX packets 74351454  bytes 125550170072 (125.5 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 432  bytes 36844 (36.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 432  bytes 36844 (36.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tailscale0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1280
        inet 100.96.250.21  netmask 255.255.255.255  destination 100.96.250.21
        inet6 fd7a:115c:a1e0::7101:fa15  prefixlen 128  scopeid 0x0<global>
        inet6 fe80::b0f7:5995:a771:74c1  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 2809  bytes 244135 (244.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2095  bytes 3034235 (3.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
ok so apparently the ping process is not processing incoming ICMP replies.. that's why mtu of 1296 is returned from the script.

x > 8.8.8.8: ICMP echo request, id 9, seq 1, length 1480
8.8.8.8 > x: ICMP echo reply, id 9, seq 1, length 76
x > 8.8.8.8: ICMP echo request, id 9, seq 2, length 1480
8.8.8.8 > x: ICMP echo reply, id 9, seq 2, length 76
x > 8.8.8.8: ICMP echo request, id 9, seq 3, length 1480
8.8.8.8 > x: ICMP echo reply, id 9, seq 3, length 76

Code:
root@test:~# ping -A -c 3 -M do -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1472(1500) bytes of data.

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2003ms

anyone knows why?