@neodemus You are my hero for linking to that post. This was driving me insane.
To all those future internet strangers out there, it's jumbo frames. Disable them and you should be back to normal.
It's curious that I was using a cheap no name switch previously and it was working fine but I just upgraded my rack and put in a Cisco switch to learn IOS and it wouldn't work with it. I would think it would have been the other way around.
Jumbo frames could be important to network performance, depending on what you do with your Proxmox. I would disable them only if MTU on the switch cannot be increased. I had the same problem years ago on a Dell blade chassis M1000e with Dell 10/40 MXL switches (ex-Force10), but I easily fixed it by increasing MTU to 12000 on trunk interfaces Proxmox blades were connected to. Not that every switch allows MTU to be set that high, but it should be possible to set it at least to 9000.
The problem is - different platforms/oses, different vendors and even different devices from the same vendor use the MTU setting ambiguously. In reality, there are multiple MTUs in play, most notably, a layer 2 MTU that applies to any ethernet frame (or any other L2 protocol) and a layer 3 MTU, so called "IP MTU", that applies to IP packet size, including the IP header. This is why, for example, you can set MTU to 9216 on a trunk port on a Cisco IOS (XE) router and then set "ip mtu 1500" on its L3 subinterface. If you don't set "ip mtu", the subinterface will use main interface's MTU. The same goes for, say, Juniper devices where you can set mtu on interface level and then separately for each family section. Why is this important ? Because a single interface can carry more than just IP traffic and while it is ok for IP traffic to have MTU 1500, other types of traffic may require more, due to different defaults, encapsulation, etc. For example, for an LDP-based MPLS pseudowire to pass, L2 MTU of an interface has to be at least 1522 bytes (1514 encapsulated untagged ethernet frame + 4 bytes VC tag + 4 bytes outer tag), but more likely 1530 (1518 encapsulated tagged ethernet frame + 4 bytes control word + 4 bytes VC tag + 4 bytes outer tag). Yet the IP MTU of that same interface can (and probably should) be explicitly set to 1500. Many more follow the same pattern, but not all. I haven't been playing with Linux networking all that much, but AFAIK, on Linux, you cannot set L2 MTU and IP MTU on the same interface separately - only the interface (L2) MTU, but you can use subinterfaces for different purposes, each with it's own MTU.
In normal circumstances, the default IP MTU is 1500 on ethernet networks for just about any device, so setting it higher on public/internet facing interfaces would lead to problems - the least of which being decreased performance and increased load due to fragmentation. However, on isolated segments, like, for example, a dedicated Proxmox cluster network or network between Proxmox nodes and a NAS/SAN device, using high IP MTU is recommended for performance. This involves setting IP MTU on all interfaces connected to that segment, but also (L2) MTU on switches those interfaces are physically connected to. Now, there's another ambiguity - some platforms expect raw MTU values (including the frame header size), some just the payload size (making you account for the header size yourself when calculating the payload size in relation to the L2 MTU of the switch). For example, IOS XR devices expect raw L2 MTU on l2transport interfaces, which would normally be 1514 (1500 bytes default payload size + 14 bytes ethernet frame header) on untagged interfaces and 1518 on tagged subinterfaces (1500 bytes default payload size + 14 bytes header + 4 bytes 802.1q tag), but on L3 (sub)interfaces, they expect just the payload size, which would normally be 1500.
My point is - when MTU is concerned, there is no "one size fits all" approach. You have to get to know the devices you are working with and understand how end-to-end MTU across different devices reflects on allowed packet sizes and overall performance, or you have to experiment and hit that sweetspot by trial and error.
