10gbe Network speed nowhere close.. what am I doing wrong?

[{c}guy_123]

Hi all,
I have 2 proxmox servers each running about 3-5 VM's and 10gbe network cards between them with a 10gbe switch as well. When I run iperf3 the best I can get is around 3-4Gb/s. I understand that there is network overhead, but I think I should be a lot closer to 7-9? I've messed with all kinds of settings, and tried different configs, but no dice.

I have my primary gateway (management interface) on vmbr1 and it's regular gigabit.
All my vm's share a 10GB/s interface on vmbr0.
This is the same config on both servers.

Can someone tell me what is the recommended way to configure this... I have a VM running pFsense which also shares the 10Gb/sec vmbr0 for lan duties.

This is driving me nuts... and just FYI, I've multiple different options for iperf3 and file transfers.. .I have a 24 hard drive zfs system that should be able to saturate that 10GB/s interface no problem... but still nothing close.

Thanks in advance.

{c}guy

 

[Dunuin]

You are probably limited by how fast your CPU can handle the packets. With a 1500 MTU my 10Gbit NIC also can only achieve 3-4 Gbit. After setting the MTU to 9000 I get the full 10Gbit as packets are bigger, so the CPU needs to handle less packets for the same bandwidth.

 

[{c}guy_123]

Thanks. I switched the mtu on both of my 10 Gb interfaces to 9000, and things improved to about 4Gbits/sec.
Is this normal?
Should I get closer to 8-9? My gut tells me yes, but I'm really unsure.

Thanks in advance for any guidance.

Never mind.... I did some additional testing, and it turns out the platform makes a HUGE difference in the speeds with iperf3. On windows the best I can get it about 3-4 Gbit/sec. but on debian --> debian (both VM's running on different servers ) here's the results:

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 216 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver

I'd say that's about right!

 

[Dunuin]

Are you using virtio or E1000 NICs wih your Win VMs? E1000 is fully emulated, so its slower than virtio which is paravirtualized. With Linux virtio will work out of the box, for Win you need to install the drivers when installing Win.

 

[{c}guy_123]

Yeah, I’m using virtio Ethernet. And even changed the mtu on it to 9000. Windows recognizes it as a 10gb device but I don’t get speeds anywhere near that on a Windows box. But I do on other non Windows vm’s on the same machine.

 

[mvs]

You can try to enable multiqueue in advanced settings of virtio adapter. With 4 queues we were able to saturate 10GbE 1500Bytes MTU link in our lab with Linux VMs.

 

[{c}guy_123]

You can try to enable multiqueue in advanced settings of virtio adapter. With 4 queues we were able to saturate 10GbE 1500Bytes MTU link in our lab with Linux VMs.

Thanks I'll give that a try. Is ethtool a requirement to set up multiqueue?

 

[mvs]

Ethtool is not required for Linux guests. Virtio-net driver will get correct setting on its own.

 

[{c}guy_123]

Ethtool is not required for Linux guests. Virtio-net driver will get correct setting on its own.

ok, so where do I enable multiqueue on Proxmox host? I only see documentation using ethtool. Also I'm running windows guests, so is the multiqueue setting in the advanced section of the driver?

 

[mvs]

You can enable it in Proxmox GUI :
VM -> Hardware -> NIC -> Advanced -> Multiqueue
It should not be larger then vCPU count of the VM

 

[{c}guy_123]

You can enable it in Proxmox GUI :
VM -> Hardware -> NIC -> Advanced -> Multiqueue
It should not be larger then vCPU count of the VM

OK I see that.... that's for the VM side, is there anything to do on the host? and when you say vCPU does that mean cores or actual cpu's assigned? I have a VM with a single CPU but 8 cores assigned. So should I set mutiqueue to 8?

Thanks for your help. Really appreciate it.

 

[mvs]

No, you do not need to set anything for your host.
vCPU is a virtual CPU Core, so you can set multiqueue up to 8 (perhaps try 4 first).

 

[{c}guy_123]

No, you do not need to set anything for your host.
vCPU is a virtual CPU Core, so you can set multiqueue up to 8 (perhaps try 4 first).

Thank you! That made all the difference. Went from about 500mb to 4Gb on one Windows 10 vm machine.

 

[grpanho]

Hi,

I have the same problem, and after setting MTU to 9000 (or lower, tested with 5000) on the virtual NIC of the VM and on the pfSense interface, the connection between Proxmox and pfSense achieves gigabit speeds. However, the time it takes for the connection to be established gets significantly slower, and after a few seconds, pfSense starts blocking literally everything, even LAN.

This behavior only happens after rebooting pfSense. If I set the MTU through ifconfig, it works.

Any idea why this happens? There is nothing on the firewall logs indicating the reason, and after lowering the MTU to 1500 and rebooting, things normalize.