Hello all,
I have been testing the SDN network feature and successfully set up a EVPN zone using the steps outlined in the documentation: https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_zone_plugin_evpn
Everything seems to be working. But the speed I am getting between hosts is a bit slower than I expected. I am using bonded 25G mellanox adapters, that should have VXLAN offloading so I would think they should reach similar speeds as with VLANs.
Does anyone have experience with this. Are these speeds normal? Or, is there something that I am missing?
Here is the speed I am getting between two VMs with a single thread with only VLAN encapsulation:
On the same two VMs I am getting a bit less
As far as I can tell, all offloading options that might matter for VXLAN are enabled:
I have been testing the SDN network feature and successfully set up a EVPN zone using the steps outlined in the documentation: https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_zone_plugin_evpn
Everything seems to be working. But the speed I am getting between hosts is a bit slower than I expected. I am using bonded 25G mellanox adapters, that should have VXLAN offloading so I would think they should reach similar speeds as with VLANs.
Does anyone have experience with this. Are these speeds normal? Or, is there something that I am missing?
Here is the speed I am getting between two VMs with a single thread with only VLAN encapsulation:
# iperf -c 172.24.210.239
------------------------------------------------------------
Client connecting to 172.24.210.239, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 172.24.210.154 port 49416 connected with 172.24.210.239 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0167 sec 8.03 GBytes 6.89 Gbits/sec
On the same two VMs I am getting a bit less
~# iperf -c 10.22.22.1
------------------------------------------------------------
Client connecting to 10.22.22.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 10.22.22.3 port 36520 connected with 10.22.22.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0251 sec 4.81 GBytes 4.12 Gbits/sec
As far as I can tell, all offloading options that might matter for VXLAN are enabled:
~# ethtool -k bond0
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [requested on]
tx-esp-segmentation: off
tx-udp-segmentation: on
tx-gso-list: off [requested on]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off
esp-tx-csum-hw-offload: off
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]