10 Gbps issue with pfSense on Proxmox

[Nyxtorm]

Hello,

I've already posted on the Netgate forum on the same subject, but I'll try here if the problem is with Proxmox.

I'm currently trying to understand a strange behavior with my pfSense CE 2.6 virtualized on Proxmox VE 7.4-3. I've been tearing my hair out for days now.

My ISP router (Freebox Delta w/ 10G-EPON, Free ISP in France) is wired directly to my Proxmox hypervisor via a passive 10Gbps DAC.

So I have a Linux bridge (vmbr2) connected to the corresponding SFP+ port.

Attached to this bridge are my pfSense WAN and two test VMs. For each of my tests, I download directly from my ISP's router, which is capable of generating data on the fly to test local speeds.

I put my tests in Pastebin: https://pastebin.com/raw/qxAUGynQ

Proxmox handles the 10Gbps link very well, since I can reach them without any problem as soon as I stop going through pfSense. I've also run a number of speedtest, which show the same behavior towards external servers: speeds seem to be "throttled" as soon as I go through my pfSense virtual machine (clean installation with no modifications).

I thought it was a FreeBSD limitation, but a nude FreeBSD installation perfectly exploits the 10Gbps link.

So I'm a bit lost, I'm throwing a bottle into the sea in case someone has the same experience and, more importantly, the solution. I'd like to avoid having to buy a Netgate 6100 when my current equipment should be sufficient.

What I've already tried:

• Test all possibilities with Hardware Checksum Offloading, TSO and LRO
• Test with or without PCIe passthrough on pfSense, maybe a small difference but really not a big deal
• Reinstall clean pfSense (test with CE 2.6, Plus 23.01 and Plus 23.05 versions)
• Test of E1000, Realtek and VMX drivers
• Test with multiqueue (4 or 8) with 8 vCPU, no difference and I can reach 10Gbps with only 2 vCPU and without multiqueue without any problem.
• Test with OPNsense, throughputs significantly higher, but no major difference.
• Test with i440fx and q35 machine
• Test with Jumbo frames (MTU 9000)
• Test new VM Untangle NG Firewall (Arista) : I'm able to use 10Gbps, including through the router, but... there is a bug if IPv6 gateway is on fe80::/10, no default route created)
• Test new VM pfSense with OS set to "Other" and UEFI, same problematic behavior

My config :

• ISP :
  • Freebox Delta 10G-EPON 8Gbps/700Mbps
• Hypervisor :
  • Proxmox VE 7.4-3
  • AMD Ryzen 7 5700G
  • 64 GB RAM
  • Motherboard MAG B550M MORTAR WIFI
  • 1 To NVMe
  • 2 x SFP+ 1/2.5/10Gbps (BCM57810S)
  • 1 x RJ45 2.5 Gbps (RTL8125B, not used)

A brief overview of the network part concerned :

Sorry for my English and thank you for your help !

 

[spirit]

How do you bench ? (what tool, what packet size ?)

I wonder if it's not a freebsd virtio driver bug not enabling multiqueues automatically. (on linux, some years ago, we need to enable queues manually in guest os).
Also , you really need to use virtio nic. (e1000,realteak,vmware will be limited to something like 300mbit/s-700mbits)

also mulitiqueue will work only with parallel connections.

 

[androme13]

Hi, i have similar configuration (freebox delta server) but vyos and PCIe passthrough instead pfsense , could you try to turn your freebox into bridge mode instead router mode?

my resuls from one VM :
curl -L -k -o /dev/null http://mafreebox.freebox.fr//gen/10G
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 525 100 525 0 0 12747 0 --:--:-- --:--:-- --:--:-- 13125


You will have to set dhcp server and ipv6 prefix delegation in your pfsense since freebox does not provide these services in bridge mode (same for WIFI).

Please also try iperf3 test, from internet and between yours VMs.

I get average 3 Gbits/sec (average 7 the night) from internet in day
[SUM] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver

and i get about 40 Gbits/sec between one VM and my vyos firewall VM
[ 5] 0.00-10.00 sec 48.7 GBytes 41.8 Gbits/sec receiver

My hardware is AMD 3900X with 64 GB RAM and intel nic X710 (with SR-IOV, huge gains and lot of less CPU overhead).

 

[_gabriel]

try with an aria2c -x 5 -s 5 http...
to force 5 parallel connections.
(off topic, for info, here from sfr isp with their cheap 500Mb/s and the bad ipv4 cgnat, i need patched (16 connections) aria2c to download big files at excepted speed, if not within the browser with one thread, mainly limited, to ~
150Mb/s ... I remember too, my old freebox adsl need multi connections too to achieve max speed, was 6 Mb to 10 Mb à l'époque )