PCIe passthrough of Chelsio T520-BT crashes under Proxmox 8, worked in 7

[ohmantics]

I have a Q35 guest passing through a Chelsio T520-BT ethernet card. It locks up with a black screen before anything is output to the console, text or graphics.

bios: ovmf
boot:
cores: 12
cpu: host
efidisk0: local-zfs:vm-100-disk-1,efitype=4m,size=1M
hostpci1: 0000:3b:00,pcie=1,rombar=0
hostpci2: 0000:5e:00,pcie=1,rombar=0
hostpci3: 0000:5f:00,pcie=1,rombar=0
machine: q35
memory: 196608
meta: creation-qemu=7.1.0,ctime=1674451925
name: freenas
net0: virtio=CE:89:5E:8D:F3:12,bridge=vmbr1
numa: 0
ostype: other
sata0: /dev/disk/by-id/ata-32GB_SATA_Flash_Drive_E0318293800000001245,size=31266648K
sata1: /dev/disk/by-id/ata-32GB_SATA_Flash_Drive_E0318293800000001249,size=31266648K
scsihw: virtio-scsi-single
smbios1: uuid=f8796147-7449-44ec-9fd7-cee5b0f8c66a
sockets: 2
unused0: local-zfs:vm-100-disk-0
vga: std
vmgenid: a8a24189-4891-49c5-9d83-95279b100a0d

Grabbed this by adding a serial0 with "qm terminal 100":

!!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000
RIP  - 000000006E71AFA4, CS  - 0000000000000038, RFLAGS - 0000000000010206
RAX  - 7FFFFFFFFFFFFF43, RCX - 800000000000001A, RDX - 000000006E7267D0
RBX  - 000000006E7267D0, RSP - 000000007EDAF780, RBP - FFFFFFFF918D982F
RSI  - 000000000000002D, RDI - 800000000000001A
R8   - 000000000000002E, R9  - 000000006D880598, R10 - 0000000000000080
R11  - 000000006D880698, R12 - 000000006DC12298, R13 - 800000000000001A
R14  - 0000000000000000, R15 - 000000000000002E
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 000000006EA01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000006E653000 0000000000000047, LDTR - 0000000000000000
IDTR - 000000006E14C018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000007EDAF3E0
!!!! Find image based on IP(0x6E71AFA4) d:\edk4\edk2.4\myworkspace\Build\MdeModule\DEBUG_VS2008x86\X64\MdeModulePkg\Bus\Pci\ChelsioUEFI\ChelsioUD\DEBUG\ChelsioUD.pdb (ImageBase=000000006E6F7000, EntryPoint=000000006E6F72C0) !!!!

I tried swapping /usr/lib/firmware/cxgbe4/t5fw.bin back to the version that shipped with Proxmox 7 and there was no difference. The guest starts normally if hostpci3 (the Chelsio T520-BT) is removed from the config. Reverted to Proxmox 7 using ZFS boot environments and everything works.

 

[ohmantics]

Updated the firmware on the card from 1.27.0.0 to 1.27.3.0 and the problem seems to have resolved itself.

Seems like upgrades to TianoCore in Proxmox are a risk... perhaps there should be a feature to select which version of UEFI firmware is used for each VM (if there isn't already).