pfSense VM slower than expected speeds between VLANs

[mshorey]

I'm hoping someone else may have some insight for me that is running pfSense as a VM on their proxmox instance and maybe not seeing the speeds they expect between VLANs. Iperf3 testing between VMs on the same VLAN I can reach transfer speeds of ~30Gbps. But between VMs on different VLANs (when the traffic needs to be routed through the pfSense VM) I'm seeing maybe 5-6Gbps. Watching the CPU utilization on the pfSense VM when it's having to route these iperf tests between the VLANs it never goes over 15% or so. I have set multiqueue to 4 or 8 for each VM depending on their vCPU count and that hasn't seemed to make a difference. All VMs are using virtio to connect and tagging their traffic for specific VLANs. I was previously doing everything over a single linux bridge (vmbr0) but I thought I'd add a second bridge (vmbr1) to pfSense just for my VM's vlan (100) to see if that would make a difference and it did a little. Previously I was seeing about 4Gbps now I'm seeing just under 6Gbps testing from a VM on VLAN1 (vmbr0) to VLAN100 (vmbr1) but am seeing more retransmits than I'd expect. Everything on my network is still 1500mtu and I would change it if I thought it'd make a difference but the CPU utilization on my pfSense VM is incredibly low so that would not be indicative of needing to use jumbo packets IMO. I'm open to any suggestion ya'll might have and extremely appreciative. My pfSense version is 2.7.2 and the VM config is as follows:
root@pve-1:/etc/pve/qemu-server# cat 107.conf
agent: 1
balloon: 0
boot: order=scsi0;ide2
cores: 8
cpu: host
hostpci0: 0000:01:00.1,pcie=1
ide2: local:iso/pfSense-CE-2.7.2-RELEASE-amd64.iso,media=cdrom,size=854172K
machine: q35
memory: 8192
meta: creation-qemu=8.1.2,ctime=1702956906
name: PFSENSE-2
net0: virtio=BC:24:11:EF:4B:41,bridge=vmbr0,queues=8
net1: virtio=BC:24:11:89:B5:BB,bridge=vmbr1,queues=8
numa: 0
onboot: 1
ostype: l26
scsi0: VMs:vm-107-disk-0,iothread=1,size=12G
scsihw: virtio-scsi-single
smbios1: uuid=7aa0d7e5-90b6-444c-98ec-2bcdab0a0e43
sockets: 1
startup: order=1,up=30
vmgenid: 331695e6-6024-4a0f-a672-cd39aac55e20
And here's one of the testing VMs:
root@pve-1:/etc/pve/qemu-server# cat 103.conf
agent: 1
balloon: 0
boot: order=scsi0
cores: 4
cpu: host
memory: 16384
meta: creation-qemu=7.0.0,ctime=1662999076
name: DOCKER
net0: virtio=36:C7:81:594:93,bridge=vmbr0,queues=4
numa: 0
onboot: 1
ostype: l26
scsi0: VMs:vm-103-disk-0,discard=on,iothread=1,size=64G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=f2c40f91-177b-410f-ab0d-c930e7b6160f
sockets: 1
startup: order=4,up=30
tags:
vmgenid: 3abb9524-cb3e-46a2-8862-52d37bce2907
And the other testing VM:
root@pve-1:/etc/pve/qemu-server# cat 112.conf
agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=7.1.0,ctime=1672979532
name: NPM
net0: virtio=2a:ed:b7:28:34:63,bridge=vmbr1,queues=4,tag=100
numa: 0
onboot: 1
ostype: l26
scsi0: VMs:vm-112-disk-0,discard=on,iothread=1,size=42G
scsihw: virtio-scsi-single
smbios1: uuid=47b48916-70d2-4543-9220-c350461b565e
sockets: 1
vmgenid: 28600f0b-74d5-494d-a64e-c4559cfc5edc
And I've attached a screenshot of the iperf3 testing performed after making the changes mentioned.