PVE problem upload speed VM

[nubje]

Hello,

Currently I got a weird problem that I cannot seem to solve on my own, that is why I am calling for your help.
My internet speed is 1Gbps up and down which is plugged into my own Mikrotik router which limits (queues) it to 900Mbps.

My desktop pc (wired) makes speedtest 860Mbps up and down with stable ping.
LXC debian makes speedtest 830-860 Mbps up and down with stable ping.
Ubuntu VM makes speedtest 800-860 Mbps down and 300-350 Mbps up with stable ping.
Windows 10 VM makes speedtest 800-860 Mbps down and 50-200 Mbps up with stable ping.

Basicly I got 1 vmbr with 1 SFP+ NIC (of quad Intel X710-DA), with VLAN tag on VM/LXC's.

Desktop PC is wired to router via 1 Gbit switch.
PVE vmbr is wire with 10Gbit to 10Gbit switch to 10Gbit port on router.
LXC and both VM's use the 10Gbit link. But somehow the VM's can not saturate the 1Gbit upload.

Already tried:
setting CPU to host
multiqueue to number of vCPU
virtio vNIC ofcourse but tried updating drivers in guest
proxmox up-to-date (8.2.2)

What to try next?
If you guys need more info or settings, please let me know and I will provide them.

 

[_gabriel]

try with speedtest "Single" connection.
try iperf3 between pve (host + Guests) and your Desktop

 

[nubje]

I did test, multiple and single, no difference in result.
I will try iperf3 this week.

Tested another situation because my gut feeling is telling me it is the NIC.
Got a spare NIC on the motherboard (although Realtek), plugged it in the router (1Gbit).
Made new bridge and set it up using that NIC.
Windows 10 VM change the vNIC from vmbr0 to new vmbr2.
Speedtest result 680Mbps up and down.

Which leads me to 2 possible problem sources:
1 - The Intel X710 NIC
2 - The route from the NIC to the router (transceivers/10Gbit switch)

Because the LXC does fine with upload I don't think it is the route from NIC to router. But it makes me wonder if the combination of the NIC and the virtIO vNIC is giving me trouble.

For clarification, all speedtests are done with the same destination server in my area.

 

[_--James--_]

Since you are tagging, what is doing your routing? Is your PC in the same VLAN the VMs are in?
Since PVE7.2 the windows virtIO NIC issues were resolved and I have been able to push 2G-3G on any modern windows install since. Been using x710's X520's and X540's with no issues there too. So if your VM is routing through something to get to the internet/physical devices for iperf you need to change that to get a proper test.

 

[nubje]

My router is doing the inter-VLAN routing. My desktop PC is not in the same VLAN as the VMs but the LXC is in the same VLAN. But results differ.
For completeness-sake I will test with the VMs in the same VLAN as my PC.

Next test I would like to do, in order to prove or disprove my feeling for the problem being something to do with the X710 NIC. I have a spare PVE machine with X520-DA2. I will exchange the X710 with that and repeat the speed test. I that case everything except the NIC is the same.
If the X520 is ok I leave it in for the moment so I can try to reproduce the problem on the spare PVE. There I can then try to solve it.

@_--James--_: What firmware are you using with the X710's? Do you know how I can check mine?

 

[_--James--_]

What firmware are you using with the X710's? Do you know how I can check mine?

ethtool -i enp3s0f0np0
driver: i40e
version: 6.8.4-2-pve
firmware-version: 7.10 0x800075df 19.5.12
expansion-rom-version:
bus-info: 0000:03:00.0

 

[nubje]

Thanks, haven't had time to reply untill now...

Tried updating the firmware but this went horribly wrong, bricked the NIC and exchanged it with the spare X520-da2.
Got the same result with the X520.

Somehow I managed to recover the X710's firmware got it working in my spare PVE.

Went further in the investigation and eventually disabled the queues in my router. This instantly released the full bandwidth of my WAN.
So it is not a problem of Proxmox but the combination of the VM and my router queues.
Still trying to find out what is wrong with the queues and why this combination cripples the upload speed though..