Hi, i have similar configuration (freebox delta server) but vyos and PCIe passthrough instead pfsense , could you try to turn your freebox into bridge mode instead router mode?
my resuls from one VM :
curl -L -k -o /dev/null http://mafreebox.freebox.fr//gen/10G
% Total % Received % Xferd...