Speedtest Throughput too low

[Urbaman]

Hi,

I recently updated my internet contract to 2,5G, and I can speedtest up to 0,9G with a CAT6 cable from a 2,5 switch to my 1G NIC on PC.
As soon as I connect the same cable to a transceiver on my 10G SFP+ NIC on proxmox server, I begin to get up to 0,5G.
Also, I cannot get more if I connect the same switch 10G SFP+ NIC to the same 10G SFP+ NIC on proxmox through a 10G DAC.

All other 10G SFP+ ports are working fine (both NICs from the same card and from other cards), doing 3xNIC or 4xNIC LACPs to another 10G switch. Iperf says 9G on those networks.

It seems there's something happening in there?

How can I properly engage with this?
PS: I do have a virtualized pfSense on that "wan" port.

Thank you.

 

[Urbaman]

Ok,

So it seems to be pfSense to cap the throughput, as putting the gateway directly to the ISP modem gets a 2G Throughput.

I'll try to get other experiments and get back.

 

[Urbaman]

Ok,

A new ubuntu VM with a VirtIO NIC on the vmbr0 bridge gets the 2G throughput. It's the very same settigs as the "wan" NIC in pfSense, which is getting up to 0,6G of throughput.

Has anyone had the same problem with pfSense?

 

[logan893]

Have you disabled Hardware Offloading? Make sure all three (Checksum, TCP Segmentation, Large Receive) are disabled from the System -> Advanced menu, Networking tab.

https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html
https://docs.netgate.com/pfsense/en/latest/virtualization/virtio.html

In virtualization cases such as Xen/KVM it may be necessary to disable checksum offloading on the host as well as the VM. If performance is still poor or has errors on these types of VMs, switch the type of NIC if possible.

You may also need to disable it for the NIC on the proxmox host.
https://forum.opnsense.org/index.php?topic=16531.0

Another option is to give pfSense direct access to the NIC via PCIe passthrough, if possible.

How do you test your network throughput?

 

[Urbaman]

I tested with ookla speedtest.

In the meantime I found out that the problem was suricata on pfSense: disabling it the throughput got up to 1,9G - that's in line with expected.

Is there a way to optimize suricata (I see something about multiqueue - how to manage them? Are them usable on pfSense/FreeBSD with VirtIO NICs?)

Thank you.

 

[logan893]

I tested with ookla speedtest.

In the meantime I found out that the problem was suricata on pfSense: disabling it the throughput got up to 1,9G - that's in line with expected.

Is there a way to optimize suricata (I see something about multiqueue - how to manage them? Are them usable on pfSense/FreeBSD with VirtIO NICs?)

Thank you.

Unfortunately no, no clue about Suricata more than that it's typically very CPU intensive for little to no gain for a home network (other than as a learning experience), nor how well it plays with Proxmox.

 

[Urbaman]

@logan893 I really think it's related to multiqueue. Any IDS/IPS like suricata is quite cpu intensive, and pfSense seems not to be multiqueue aware on vtnet (VirtIO), so we end up with only one queue, so just one vCpu used up by suricata, ending up with a lot of lag and performance disruption at the end.

I also can't passthrough the NIC ATM, I'll try moving around the cables a little bit to put pfSense on a iommu-isolated NIC (I'm using one of the Broadcom NICs on iommu group 57, I'll try to move one of the Intel cards), but passthrough should be the only real solution to this.

pvesh get /nodes/pvenode2/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────┬──────┬──────────────────┬───────────────────────┬─────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                                                                  │ mdev │ subsystem_device │ subsystem_device_name │ subs
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════╪═════
│ 0x020000 │ 0x16a1 │ 0000:01:00.0 │         57 │ 0x14e4 │ BCM57840 NetXtreme II 10 Gigabit Ethernet                                                    │      │ 0x1f79           │                       │ 0x10
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x16a1 │ 0000:01:00.1 │         57 │ 0x14e4 │ BCM57840 NetXtreme II 10 Gigabit Ethernet                                                    │      │ 0x1f79           │                       │ 0x10
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x16a1 │ 0000:01:00.2 │         57 │ 0x14e4 │ BCM57840 NetXtreme II 10 Gigabit Ethernet                                                    │      │ 0x1f79           │                       │ 0x10
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x16a1 │ 0000:01:00.3 │         57 │ 0x14e4 │ BCM57840 NetXtreme II 10 Gigabit Ethernet                                                    │      │ 0x1f79           │                       │ 0x10
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:04:00.0 │         60 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:04:00.1 │         61 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:04:00.2 │         62 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:04:00.3 │         63 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x154d │ 0000:81:00.0 │         17 │ 0x8086 │ Ethernet 10G 2P X520 Adapter                                                                 │      │ 0x7b11           │ 10GbE 2P X520 Adapter │ 0x80
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x154d │ 0000:81:00.1 │         18 │ 0x8086 │ Ethernet 10G 2P X520 Adapter                                                                 │      │ 0x7b11           │ 10GbE 2P X520 Adapter │ 0x80
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:8c:00.0 │         28 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:8c:00.1 │         29 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:8c:00.2 │         30 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────
│ 0x020000 │ 0x1572 │ 0000:8c:00.3 │         31 │ 0x8086 │ Ethernet Controller X710 for 10GbE SFP+                                                      │      │ 0x0000           │                       │ 0x15
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼─────

This obviously puts a strong NO to proxmox own HA on pfSense, but I'm using two pfSense in HA on two different nodes, so it should be ok to passthrough.

Time to tweak my network to get the most out of virtualized pfSense, hoping not to disrupt it!