OPNsense in Proxmox, slow WAN speed from physical LAN

[lauka]

I have Proxmox running on a machine with OPNsense as a VM, acting as the main firewall/gateway. The Proxmox host has four i226 NICs:

• one is passed through directly to the OPNsense VM. Used as WAN
• one is bridged in proxmox and passed to the OPNsense. Used as LAN

Testing using speedtest.net, the OPNsense and other VMs running on proxmox can achieve the full 1Gpbs WAN speed through the bridged connection.

The LAN port is connected to Zyxel GS1920 switch. The devices connected to the switch can only achieve 300-600 Mbps WAN speed. The weird part is that iPerf3 from the same LAN clients to OPNsense (and to the other VMs) achieves full 1Gbps speed, so the hardware and wiring is capable of the full speed of the WAN.

I don't have any traffic shaping enabled in OPNsense and only the built-in firewall rules.

VLANs are not enabled in the interfaces in proxmox /etc/network/interfaces:

auto lo
iface lo inet loopback
#iface enp5s0 inet manual
iface enp6s0 inet manual
auto enp7s0
iface enp7s0 inet manual
auto enp8s0
iface enp8s0 inet manual
iface enp9s0 inet manual
auto vmbr0
iface vmbr0 inet static
        address 192.168.1.10/24
        gateway 192.168.1.1
        bridge-ports enp7s0
        bridge-stp off
        bridge-fd 0
        dns-nameservers 192.168.1.13 1.1.1.1

Things I have tried/debugged following things with no improvements:

• passthrough the LAN NIC to the OPNsense VM.
• LACP two ports from the switch to the proxmox bridge.
• reverting OPNsense to factory defaults.
• changing to pfSense with default settings.
• disabling and enabling all the HW offload features in OPNsense
• disabling ASPM for PCIE in BIOS and in grub.
• different LAN switch (Netgear R8000)
• 2 cores to the VM + multiqueue of 2

I don't know if the issue is with the Proxmox or OPNsense/pfSense and I'm starting to be out of ideas what to try next. Would anyone have any ideas how to proceed? I'm happy to provide any additional information.

 

[bor.paw]

The problem is mostly on the router side, so if you use one WAN interface, you should disable "disable-reply-to", and you will achieve 1 Gbps speed.
Take a look at this Netgate docs, OPNsense has similar options.

https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#disable-reply-to

And to be honest, you should use the vmbr rather than passthrough NIC it will give you more flexibility and nearly the same performance.

 

[lauka]

The problem is mostly on the router side, so if you use one WAN interface, you should disable "disable-reply-to", and you will achieve 1 Gbps speed.
Take a look at this Netgate docs, OPNsense has similar options.

https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#disable-reply-to

And to be honest, you should use the vmbr rather than passthrough NIC it will give you more flexibility and nearly the same performance.

Thanks! "disable-reply-to" was unchecked by default, so I tried checking it, this did not however make any difference to the throughput.