[SOLVED] Help with GPU Passthrough with NVIDIA 1660 Super on Threadripper

[argonzalez2023]

Solved! I needed to patch the 1660 Super's rom to work with KVM.

Original Post:
I am having issues passing my 1660 super to any VM on my system. I currently have 2 GPUs on my board, the 1660 super in the main pci express slot and a 1550 super on another slot. When I pass the 1550 super, the VM starts and I see output on my monitor. However, when i pass the 1660 super, the monitor it's hooked to flashes and turns black.

System
AMD Threadripper 3960x
Gigabyte trx40 aorus pro
Nvidia 1660 Super - Main GPU
Nvidia 1650 Super - Secondary GPU

VM Config
bios: ovmf
boot: order=usb0
cores: 8
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:21:00,pcie=1,x-vga=1 <- this is the 1660 Super
ide2: none,media=cdrom
machine: pc-q35-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1679125272
name: Unraid
net0: virtio=1E:73:A6:AE:C3:EC,bridge=vmbr0
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=fb20ae6e-70ab-479c-8d55-cc4f269dc2b6
sockets: 1
usb0: host=7-4.2
vga: none
vmgenid: 75f0239b-b9b9-4408-8ec6-59ce12d0976b

dmesg | grep -i vfio

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on vfio_pci.disable_vga=1 initcall_blacklist=sysfb_init
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on vfio_pci.disable_vga=1 initcall_blacklist=sysfb_init
[    7.494006] VFIO - User Level meta-driver version: 0.3
[    7.498727] vfio-pci 0000:21:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    7.520133] vfio_pci: add [10de:21c4[ffffffff:ffffffff]] class 0x000000/00000000
[    7.564059] vfio_pci: add [10de:1aeb[ffffffff:ffffffff]] class 0x000000/00000000
[    7.564069] vfio_pci: add [10de:1aec[ffffffff:ffffffff]] class 0x000000/00000000
[    7.608049] vfio_pci: add [10de:1aed[ffffffff:ffffffff]] class 0x000000/00000000
[    7.608090] vfio-pci 0000:49:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    7.628178] vfio_pci: add [10de:2187[ffffffff:ffffffff]] class 0x000000/00000000
[    7.628206] vfio_pci: add [10de:1aeb[ffffffff:ffffffff]] class 0x000000/00000000
[    7.628215] vfio_pci: add [10de:1aec[ffffffff:ffffffff]] class 0x000000/00000000
[    7.628223] vfio_pci: add [10de:1aed[ffffffff:ffffffff]] class 0x000000/00000000
[   21.195803] vfio-pci 0000:48:00.1: Masking broken INTx support
[ 3376.890311] vfio-pci 0000:21:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
[ 3376.890342] vfio-pci 0000:21:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 3376.891921] vfio-pci 0000:21:00.0: No more image in the PCI ROM
[ 3380.507577] vfio-pci 0000:21:00.0: No more image in the PCI ROM
[ 3380.507652] vfio-pci 0000:21:00.0: No more image in the PCI ROM

 

[leesteken]

Make sure to early bind the boot GPU (for which you already use initcall_blacklist=sysfb_init) to vfio-pci, otherwise Proxmox will still touch it before the VM starts. You can also put that on the kernel line with vfio_pci.ids=1234:5678,4321:8765 (except use the correct IDs).

 

[argonzalez2023]

Make sure to early bind the boot GPU (for which you already use initcall_blacklist=sysfb_init) to vfio-pci, otherwise Proxmox will still touch it before the VM starts. You can also put that on the kernel line with vfio_pci.ids=1234:5678,4321:8765 (except use the correct IDs).

This is the context of my /etc/modprobe.d/vfio.conf file

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed,10de:2187,10de:1aeb,10de:1aec,10de:1aed disable_vga=1

To make sure, I also added these values to the kernel line as you stated, still the same issue.

quiet amd_iommu=on vfio_pci.disable_vga=1 initcall_blacklist=sysfb_init vfio_pci.ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed,10de:2187,10de:1aeb,10de:1aec,10de:1aed

Still no go. When I start proxmox, i see the first two lines and then it stops. I'm assuming this is because im not allowing proxmox to use my main GPU. When I start my VM, the screen turns black and the monitor says no input.