OK, it is something in the routing, if I host iperf3 on my VMs and put my Mac on VLAN101, I get just over 3 Gbps, so cross the VLAN is cutting the speed by half for some reason. Not nearly as much I'm losing to the WAN, but there's something there, just not sure what.
It's also interesting...