Hello! I have a minisforumn ms-01 with an x540 PCIe nic added in. It's 13900h with 64gb ram.
NICs:
1x Port x540 10ge (Connected to WAN (Fiber 5gbps synchronous speeds)
2x Port x710 motherboard embedded 10ge (1 port is physically connected to a TP-Link SG3210X-M2 by DAC)
2x i226 embedded (2.5ge)
Systems:
Proxmox 8.2.8 installed to baremetal
OPNsense 24.7.8 (Proxmox VM w/ x540 PCIe Passthrough all functions for WAN and x710 virtual function for LAN)
Ubuntu 22 (Proxmox VM w/ x710 virtual function PCIe passthrough)
Net path from ubuntu: Ubuntu VF nic -> internal NIC (I monitored the switch SG3210X-M2 and VFs are internally communicating) -> OPNsense VF nic -> OPNsense and so on.
My question/issue is when I do a speedtest from Ubuntu, I get 5gbps down and 1.5gbps up. So my upload is not reaching 5gbps like it should. Iperf tests show 20gbps Ubuntu -> OPNsense and 10gbps OPNsense -> ubuntu. This is with hardware offloading enabled ( disabled gets 10gbps and 7gbps respectively).
All links show bidirectional 10gig, all drivers and OSes are up to date. I have a 10gig router that has embedded speed tests and WAN is providing consistent 5gbps up and down to rule out ISP.
I think it has something to do with SR-IOV in proxmox. In proxmox, when I set for example: ip link set enp3s0f1 vf # rate 5300, I get 5gbps up and down, but when rate is 0 (Unlimited), its 5gbps/1.5gbps. As i increase rates from 5400 to 10000, upload gets slower, eventually going back to 1.5gbps upload at around 6500 rate.
The rate of 5300 fixes the issue, but then internal network speeds (iperf) are limited to 5300 on VFs for VMs. Any suggestions or advice to get unlimited VF rates and fixing the internet upload speeds? Thank you!
NICs:
1x Port x540 10ge (Connected to WAN (Fiber 5gbps synchronous speeds)
2x Port x710 motherboard embedded 10ge (1 port is physically connected to a TP-Link SG3210X-M2 by DAC)
2x i226 embedded (2.5ge)
Systems:
Proxmox 8.2.8 installed to baremetal
OPNsense 24.7.8 (Proxmox VM w/ x540 PCIe Passthrough all functions for WAN and x710 virtual function for LAN)
Ubuntu 22 (Proxmox VM w/ x710 virtual function PCIe passthrough)
Net path from ubuntu: Ubuntu VF nic -> internal NIC (I monitored the switch SG3210X-M2 and VFs are internally communicating) -> OPNsense VF nic -> OPNsense and so on.
My question/issue is when I do a speedtest from Ubuntu, I get 5gbps down and 1.5gbps up. So my upload is not reaching 5gbps like it should. Iperf tests show 20gbps Ubuntu -> OPNsense and 10gbps OPNsense -> ubuntu. This is with hardware offloading enabled ( disabled gets 10gbps and 7gbps respectively).
All links show bidirectional 10gig, all drivers and OSes are up to date. I have a 10gig router that has embedded speed tests and WAN is providing consistent 5gbps up and down to rule out ISP.
I think it has something to do with SR-IOV in proxmox. In proxmox, when I set for example: ip link set enp3s0f1 vf # rate 5300, I get 5gbps up and down, but when rate is 0 (Unlimited), its 5gbps/1.5gbps. As i increase rates from 5400 to 10000, upload gets slower, eventually going back to 1.5gbps upload at around 6500 rate.
The rate of 5300 fixes the issue, but then internal network speeds (iperf) are limited to 5300 on VFs for VMs. Any suggestions or advice to get unlimited VF rates and fixing the internet upload speeds? Thank you!
Last edited: