Another 10Gb Networking Issue

[gilbertgrape87]

Background: not the most proficient Linux user, but enough to be dangerous. The attached picture is my configuration and below are my important data points:

• Configuration
  • TrueNAS Core shell configuration:
    • Interface=vtnet0
    • MTU=9000
    • IP = 10.2.1.27
  • VM Network Configuration:
    • Net0: virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1,mtu=9000,queues=8 (I have 8 cores)
  • Proxmox Host Configuration:
    • Vmbr0
      • Bridgeport = ens5f1
      • MTU=9000
      • IP=10.2.1.101
    • Ens5f1
      • MTU=9000
  • USW-48-Pro Configuration
    • Properly negotiates 10Gb
    • Jumbo Frames allowed
    • same vlan/subnet for all devices involved
  • UNAS-Pro
    • Properly negotiates 10Gb
    • IP=10.2.1.132
• Tests
  • Iperf
    • TrueNAS <-> UNAS = 9.1Gb/s
    • Proxmox <-> UNAS = 9.1Gb/s
  • Fio (posixaio engine)
    • TrueNAS to TrueNAS local
      • 728MB/s
    • UNAS to UNAS local
      • 330MB/s
    • Proxmox host to TrueNAS nfs share
      • 152MB/s
    • Proxmox host to UNAS nfs share
      • 75MB/s
    • UNAS to TrueNAS nfs share
      • 139MB/s

Based on the above, I feel i should be able to get to about 330GB before i start crying to Ubiquiti over the poor write speeds on their NAS. Until then, this must be a networking configuration issue. Any ideas of things to check that I haven't already checked?

Thanks