Chelsio/pfSense SR-IOV Passthrough Issue

[johnoatwork]

Hi All,

Hoping someone can tell me the obvious thing that I'm missing!

I'm running dual Chelsio T520-CRs in a Dell R620 with PVE 7.1. I have a pfSense VM on the host and am passing through the T520's.

All works fine if I pass through the whole card, but if I pass through an SR-IOV VF, pfSense just ignores it and doesn't load the driver (pciconf -lv shows the Chelsio cards as present but driver is none).

I've pretty much ruled out everything except driver/firmware compatibility. When Proxmox loads it checks the firmware version on the T5 and updates it to 1.26.2.0 if it is less than this. If I pass the whole card through to pfSense, it downgrades the firmware to 1.25.0.40 for compatibility with its driver.

After the VM with the full card passed through has booted, the card is no longer available to the host until its rebooted and then of course Proxmox reimages the cards with 1.26.2.0.

Its pretty clear this is a pfSense issue. I've tested with OPNsense, VyOS, FreeBSD 12.2 and TNSR and they all work fine with the VFs passed through. I've also used ethtool to downgrade the firmware on the PVE host after boot but after that the pfSense instance with the VFs won't even start.

I've posted a question on the pfSense site, but I was wondering how to stop Proxmox from updating the firmware on boot? As I understand it, the firmware is included as a blob in the cxgbe4 driver, so I'm guessing this requires downgrading the driver?

Thanks for any tips!

 

[johnoatwork]

Just to update this. No solution with the Chelsio cards, but SR-IOV passthrough woks fine with Intel X710-DA2. So, noting again that SR-IOV passthrough works fine with native FreeBSD, OPNsense and Linux distros, this appears to be a pfSence issue with the Cheliso cards. I've given up for the moment and am just passing the whole card through.

 

[vesalius]

Wonder if this is an issue or regression with FreeBSD 13 (the pfSense base) versus FreeBSD 12.* (the hardenedBSD OPNsense base).

 

[johnoatwork]

Wonder if this is an issue or regression with FreeBSD 13 (the pfSense base) versus FreeBSD 12.* (the hardenedBSD OPNsense base).

Actually I think pfsense is build on 12.2 STABLE.

 

[vesalius]

Actually I think pfsense is build on 12.2 STABLE.

you are correct, it will be 2 versions from now per their message boards before pfsense plus (not sure how long CE will trail behind) that is supposed to be 13. OPNsense will get there first as they plan to shift to freebsd 13 in January/feb.

Although netgate employees get a little touchy on this and state that everything good/required from current FreeBSD releases is backported to their variant when I have asked them about this in the past.

https://docs.netgate.com/pfsense/en/latest/releases/versions.html