Virtualization of 10G Nic for Pfsense

[DocHodges]

Hey guys I heard so much about proxmox I wanted to take the plunge and move some things off my unraid server and into proxmox. The first thing I wanted to do is move pfsense from a bare metal to VM. I'm a bit new with proxmox so I apologize in advance.

I have the VM created following the pfsense guide. Really no issues there. Its up an running with 2 Linux bridges linked to my Intel X550 nic ports.

The Issue I am running into is the Main network device enp1s0f0 is connected to WAN and enp1s0f1 is connected to LAN this correlates to vmbr1 and vmbr2 respectively. PfSense shows the link speed to be 10Gbase-T. This works fine for my LAN as it is connected to a 10G switch. The WAN on the other hang is connected to a comcast modem with a 2.5G port. On bare metal using the same nic I get around 2300 down for reference. Using the paravirtulaized nic I am pulling 900. I did some reading and found ethtool. It confirmed it is negotiating at 1000Mb/s. I then forced it to 2500Mb/s and can see the Speed reported in ethtool show 2500. But in this setup I only get around 275 down. Changing it back to 1000 jumps right back to the 900. I did try to change it to 5000 and 10000 and it sent the request but the speed remained at unknown as I guess it couldn't communicate. I then tried to play with the vmbr1 bridge but it seems since this is a child of the enp1s0f0 there is no changing that can be done. I would really like to get the system back up and running full speed but this is kicking my butt. Anyone have any advice or ideas on what to try next? I've spent an embarrassing amount of time fighting this before coming and posting so don't be too hard on the new guy haha. Below you will find some logs from the console of me trying to use ethtool. and despite the below not showing it I have tried to include autoneg off and it doesnt seem to take it so I jsut left it off.

Other infor that might be useful other than the X550 Nic I am using the motherboard Nic for Proxmox access and it I believe is an intel 225 nic so limited to 1gb. I am not sure if this is causing an issue either. I wouldn't think so but it might I guess.

root@pve1:~# ethtool -s enp1s0f0 speed 2500 duplex full
root@pve1:~# ethtool enp1s0f0
Settings for enp1s0f0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 2500baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 2500Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes






root@pve1:~# ethtool -s enp1s0f0 speed 1000 duplex full
root@pve1:~# ethtool enp1s0f0
Settings for enp1s0f0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

 

[DocHodges]

Nobody has any ideas on what to try?

 

[hustlebird]

This doesn't address your issue directly, but have you looked into PCIe pass through for the WAN NIC? That was something I was considering for my home-lab, so that the firewall VM could function as the internet router, and simplifying my VM network configurations.

Otherwise, there's quite a bit about 10gbps vbridges already on the forums you might try to apply. I'm unfortunately not too familiar with the autonegotiating 2.5/5/10 gbase-t ports, so I don't have any specific insights there. You are using the same cable from when you had tested bare metal right? It sounded like you might be getting errors trying to force the higher speed.

 

[Neobin]

I can unfortunately not help you with your speed problem; but something regarding the auto-negotiation with 2.5 (and 5) Gb/s on the X550:
Ootb, they do not advertise those speeds; from the ixgbe driver readme:

By default, devices based on the Intel(R) Ethernet Controller x550 do not
advertise 2.5 Gbps or 5 Gbps. To have your device advertise these speeds, use
the following:

# ethtool -s <ethX> advertise N

Where N is a combination of the following.
100baseTFull   0x008
1000baseTFull  0x020
2500baseTFull  0x800000000000
5000baseTFull  0x1000000000000
10000baseTFull 0x1000

For example, to turn on all modes:
# ethtool -s <ethX> advertise 0x1800000001028

For more details please refer to the ethtool man page.

NOTE: On Linux systems with INTERFACES(5), this can be specified as a pre-up
command in /etc/network/interfaces so that the interface is always brought up
with NBASE-T support. For example:

# iface <ethX> inet dhcp
    pre-up ethtool -s <ethX> advertise 0x1800000001028 || true

https://downloadmirror.intel.com/738735/readme.txt