Speed problem with MTU 9000 hypervisor and MTU 1500 VMs

[AlexandreAngel]

Hello,
We have speed loss with mtu set to 9000 and WMs using a mtu of 1500.

Here is the configuration, we have 2 proxmox connected to a switch :

vmbr1 is an ovs bridge with mtu 9000.
Bond0 is an ovs bond with mtu 9000, each member of the bond has a mtu of 9000 using pre-up, and is link vmbr1.

An intport ovs is set with mtu 9000 to connect to our SAN. That part works fine.

VMs are connected to vmbr1 and have a mtu of 1500.

I noticed that the speed was slower after we did switch to mtu 9000.

We have a 10-20% speed loss of traffic.

To make the test, i switched back one of the server to the config with mtu 1500.

I did the test with iperf with one VM on a hypervisor and a server outside of this cluster :

With hypervisor with mtu 9000
[ 5] 0.0-20.0 sec 1.85 GBytes 795 Mbits/sec
With hypervisor with mtu 1500
[ 5] 0.0-20.0 sec 2.08 GBytes 893 Mbits/sec

I did some capture on the VM with both settings.

On the 9000 setting, the packet lenght stay between 1448 and 2896.
On the 1500 setting, the packet lenght goes to 13000-15000.

Any thought ?

 

[Alwin]

Any thought ?

Fragmentation, incoming packets with mtu 9000 have to be split up to fit the mtu 1500.

With hypervisor with mtu 9000
[ 5] 0.0-20.0 sec 1.85 GBytes 795 Mbits/sec
With hypervisor with mtu 1500
[ 5] 0.0-20.0 sec 2.08 GBytes 893 Mbits/sec

This can be also a variation, as any other network traffic will influence bandwidth.

On the 1500 setting, the packet lenght goes to 13000-15000.

This doesn't seem right, too many zero's.

 

[AlexandreAngel]

This can be also a variation, as any other network traffic will influence bandwidth.

I did the test 10 times. always the same result, the network usage on the plateform is near zero.
I tried with different VMs and real servers.

This doesn't seem right, too many zero's.

See for yourself, here is the screen shots, captures are done on the VM.

Mtu 1500, normal speed :

Mtu 9000, speed is reduced :

It looks like with mtu set to 9000, something in OVS is preventing the vm to negotiate the mss of his choice.

 

[Alwin]

This could be a explanation for the higher length then mtu.
https://osqa-ask.wireshark.org/questions/60713/mtu-size-and-packet-length-in-bytes-difference

 

[AlexandreAngel]

Ok for the explanation, but i'm just trying to understand what is happening and why do i loose speed.

I did capture on the physical server with hypervisor mtu at 1500 and 9000, the lengh is changing like on the screen shot where mtu is at 1500. The strange thing is that with mtu at 9000 on hypervisor, the lengh on the vm capture is really stable under 1500.

I did another test changing the source and the destination :

Mtu 1500 :
[ 3] 0.0-10.0 sec 2.19 GBytes 1.88 Gbits/sec
Mtu 9000 :
[ 3] 0.0-10.0 sec 2.03 GBytes 1.74 Gbits/sec