VXLAN with low bitrate transfer even with 10G SPF.

Hi... Hope everybody is ok!
I had have noticed something weird... Or, perhaps, is just the way it is...
I have 3 nodes, with 3 nic, 2x 1g and 1x 10G.
So I went ahead and create a vxlan and a zone like that:

Code:
pve101:/etc/pve/sdn# cat vnets.cfg
vnet: vxnet1
        zone vxzone
        tag 100000
        vlanaware 1

pve101:/etc/pve/sdn# cat zones.cfg
vxlan: vxzone
        ipam pve
        mtu 9000
        peers 172.18.0.20 172.18.0.30

The IPs 172.18.0 are using the 10G SPF nic. In this IP there is only cluster communications, which, I believe, is very light.

The MTU 9000 is not set etiher in the physical NIC or in the switch!
But only with MTU 9000 in the SDN, and MTU 8950 in the VM NIC configuration (virtio) I could get 3G of transfer using iperf3 as a test between the 2 vms.
Without MTU 9000/8950, i.e., using the default 1500 I could get only 1.2, 1.3 G.
My question is why?
Is there something in the SDN that hold the speed of the nics?
Sorry if I dont do myself clear enought.

Any help will do!
Thank you

Best regard

With MTU 1500
1757336807772.png

With MTU 9000
1757336824839.png
 
Last edited:
Does iperf directly between the two IPs show the same result? Does your NIC support VXLAN offloading / is it enabled?
 
Does iperf directly between the two IPs show the same result?
Hi.. Thanks for reply.
This is the result of iperf on the real nic, on the hosts:
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 172.18.0.10, port 33718
[ 5] local 172.18.0.20 port 5201 connected to 172.18.0.10 port 33728
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.09 GBytes 9.38 Gbits/sec
[ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 4.00-5.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 7.00-8.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Does your NIC support VXLAN offloading / is it enabled?
Do you mean this ethtool -k <NIC> tx-udp_tnl-segmentation on tx-udp_tnl-csum-segmentation on command?
I am not sure.
This is on the hosts, right?
According to the bellow command, it is "on" on both hosts:

pve101:~# ethtool -k ens1 |grep -i udp
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-udp-segmentation: on
rx-udp_tunnel-port-offload: off [fixed]
rx-udp-gro-forwarding: off
pve101:~# ssh pve102 ethtool -k ens1 |grep -i udp
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-udp-segmentation: on
rx-udp_tunnel-port-offload: off [fixed]
rx-udp-gro-forwarding: off
pve101:~#

Thanks for the help.
 
Can you try setting the MTU to 1450 for the VXLAN zone and the guests? (instead of 1500) Since VXLAN adds an overhead so I'd assume the host needs to fragment the packets before transferring and the receiving end needs to reassemble. Maybe that's already the solution.
 
Can you try setting the MTU to 1450 for the VXLAN zone and the guests? (instead of 1500) Since VXLAN adds an overhead so I'd assume the host needs to fragment the packets before transferring and the receiving end needs to reassemble. Maybe that's already the solution.
As I state before, with 8950 I got around 3 G/s, but with 1450 the bitrate transfer drop to around 1 G/S!

1757417745556.png
 
As I state before, with 8950 I got around 3 G/s, but with 1450 the bitrate transfer drop to around 1 G/S!
Ah, sorry I misinterpreted your initial post that you used 1500 on the zone rather than 1450. Have you double-checked that MTU is properly set up everywhere? VXLAN generally is quite sensitive to fragmentation. You could use e.g. Wireshark along the path / at the receiving end to see if there are any fragmented packets.

You can also retry with the -P X flag for iperf, to check if there aren't any potential issues with e.g. irq balancing due to there being only 1 VXLAN flow (even though the 9000 MTU result indicates differently).


Do you mean this ethtool -k <NIC> tx-udp_tnl-segmentation on tx-udp_tnl-csum-segmentation on command?
Yes, according to [1] those are the tunables that can be used for VXLAN traffic specifically - but it seems to be correctly set up in your case.

[1] https://www.kernel.org/doc/Documentation/networking/vxlan.rst