GPU Passthrough - Error 43

[sashi]

Hello
Having quite a hard time trying to passthrough my GPU to Windows. Error 43. Searched far and wide and still no luck

Host Specs
Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
NVIDIA NVS310
16GB DDR4
Samsung EVO SSD as boot
Couple of ZFS pools

PVEVERSION
pve-manager/8.1.3/b46aac3b42da5d15 (running kernel: 6.5.11-4-pve)

My grub line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 initcall_blacklist=sysfb_init"

Modprobe Blacklist.conf
blacklist nouveau blacklist nvidiafb

vfio.conf
options vfio-pci ids=10de:107d,10de:0e08 disable_vga=1

lspci | grep VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GF119 [NVS 310] (rev a1)

VM Config
balloon: 0 bios: ovmf boot: order=scsi0;ide0;ide2;net0 cores: 4 cpu: host,flags=+hv-evmcs;+aes efidisk0: data:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M hostpci0: 0000:03:00,pcie=1 machine: pc-q35-8.1 memory: 2048 meta: creation-qemu=8.1.2,ctime=1705290033 name: windows net0: virtio=BC:24:11:FE:EB:52,bridge=vmbr0 numa: 0 ostype: win10 scsi0: data:vm-101-disk-1,cache=writeback,iothread=1,size=32G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=8442112e-aff2-4e37-b845-f73a5da0dcb1 sockets: 1 vmgenid: a8bbf3e3-a80a-40c9-8ef0-9b1779edb598

IOMMU Group
IOMMU Group 32: 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [NVS 310] [10de:107d] (rev a1) 03:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)

 

[sashi]

I did some digging around dmesg and found

[ 0.319441] pci 0000:03:00.0: BAR 1: assigned to efifb [ 0.354951] pci 0000:02:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 0.354955] pci 0000:03:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window [ 0.354984] pci 0000:00:1f.3: BAR 0: assigned [mem 0xfb330000-0xfb3300ff 64bit] [ 0.355019] pci 0000:02:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 0.355021] pci 0000:02:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 0.355044] pci 0000:03:00.0: BAR 6: assigned [mem 0xfb080000-0xfb0fffff pref]

Also found some APCI errors. BIOS has been updated

[ 14.146782] ACPI Error: Needed [Buffer/String/Package], found [Integer] 0000000013318e1d (20230331/exresop-557) [ 14.146796] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.146829] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.146838] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.147637] ACPI Error: Needed [Buffer/String/Package], found [Integer] 000000000bd57af4 (20230331/exresop-557) [ 14.147648] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.147679] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.147688] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.148521] ACPI Error: Needed [Buffer/String/Package], found [Integer] 000000001340bf01 (20230331/exresop-557) [ 14.149609] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.149649] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.149660] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.150791] ACPI Error: Needed [Buffer/String/Package], found [Integer] 00000000d3639a0a (20230331/exresop-557) [ 14.150800] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.150831] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.150840] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529)

 

[leesteken]

[ 0.319441] pci 0000:03:00.0: BAR 1: assigned to efifb

This makes me think you did not apply the initcall_blacklist=sysfb_init work-around (which you claim you did).
What is the output of cat /proc/cmdline?

Also found some APCI errors. BIOS has been updated

[ 14.146782] ACPI Error: Needed [Buffer/String/Package], found [Integer] 0000000013318e1d (20230331/exresop-557) [ 14.146796] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.146829] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.146838] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.147637] ACPI Error: Needed [Buffer/String/Package], found [Integer] 000000000bd57af4 (20230331/exresop-557) [ 14.147648] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.147679] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.147688] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.148521] ACPI Error: Needed [Buffer/String/Package], found [Integer] 000000001340bf01 (20230331/exresop-557) [ 14.149609] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.149649] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.149660] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.150791] ACPI Error: Needed [Buffer/String/Package], found [Integer] 00000000d3639a0a (20230331/exresop-557) [ 14.150800] ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [Index] (20230331/dswexec-433) [ 14.150831] ACPI Error: Aborting method \_SB.WMIV.WVPO due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529) [ 14.150840] ACPI Error: Aborting method \_SB.WMIV.WMPV due to previous error (AE_AML_OPERAND_TYPE) (20230331/psparse-529)

Ask the motherboard manufacturer to fix this (but they will probably say they don't support Linux).

 

[sakkenix]

I have had the same issue in the past.
Some GPUs have troubble unless you manually specify the BIOS of the GPU when you pass it trough.
Here are the steps you need to take to dump BIOS and passtrough.

1. Dump BIOS
   Make sure you "tab-complete" path to make sure you get the right one.
   Dumping the rom file should be compelty safe, but can cause permanent damage to the GPU! If the option below don't work for you, try to extract the rom using GPU-Z on a windows machine.
   

   cd /sys/bus/pci/devices/0000:03:00.0/
   echo 1 > rom
   cat rom > /usr/share/kvm/gpu-NVS310.bin
   echo 0 > rom

2. In the VM config file, change the hostpci0 line to the following to include the BIOS file.
   hostpci0: 0000:03:00,pcie=1,romfile=gpu-NVS310.bin

Some NVIDIA cards might experience more issues with the rom file itself, and you might need to patch it. Take a look at the following projects if you experience more issues. https://github.com/sk1080/nvidia-kvm-patcher https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher. Or you can do it yourself wiht a HEX editor.

Good luck

 

[sashi]

This makes me think you did not apply the initcall_blacklist=sysfb_init work-around (which you claim you did).
What is the output of cat /proc/cmdline?

Ask the motherboard manufacturer to fix this (but they will probably say they don't support Linux).

I added a bunch of stuff hoping it would help but it didn't but yes I do have that in there

cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-4-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init video=vesafb:off video=efifb:off vga=off video=simplefb:off video=vesa:off disable_vga=1

 

[sashi]

I have had the same issue in the past.
Some GPUs have troubble unless you manually specify the BIOS of the GPU when you pass it trough.
Here are the steps you need to take to dump BIOS and passtrough.

1. Dump BIOS
   Make sure you "tab-complete" path to make sure you get the right one.
   Dumping the rom file should be compelty safe, but can cause permanent damage to the GPU! If the option below don't work for you, try to extract the rom using GPU-Z on a windows machine.
   

   cd /sys/bus/pci/devices/0000:03:00.0/
   echo 1 > rom
   cat rom > /usr/share/kvm/gpu-NVS310.bin
   echo 0 > rom

2. In the VM config file, change the hostpci0 line to the following to include the BIOS file.
   hostpci0: 0000:03:00,pcie=1,romfile=gpu-NVS310.bin

Some NVIDIA cards might experience more issues with the rom file itself, and you might need to patch it. Take a look at the following projects if you experience more issues. https://github.com/sk1080/nvidia-kvm-patcher https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher. Or you can do it yourself wiht a HEX editor.

Good luck

Thanks for this. I've tried this before but with no success. Interestingly enough though I booted a live linux ISO just for fun and lspci shows the NVS310 just fine. I didn't see any errors either.
Windows sees the NVS310 but shows the error 43

 

[leesteken]

Thanks for this. I've tried this before but with no success. Interestingly enough though I booted a live linux ISO just for fun and lspci shows the NVS310 just fine. I didn't see any errors either.
Windows sees the NVS310 but shows the error 43

At least you know it's not a passthrough issue but a WIndows or NVidia driver problem. This might help guide the search for a known work-around.

 

[sashi]

So I found an old WX3100 Radeon Pro. I grabbed it and without much config it works. Not sure what the issue with the Nvidia drivers

 

[sakkenix]

Have you looked at the proxmox PCI Passtrough wiki?
https://pve.proxmox.com/wiki/PCI_Passthrough#Error_43

Also the Arch wiki, while not 100% applicable to proxmox might help:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

 

[Joris L.]

i wasted a lot of time on pci-passthrough with AMD GPU
eventually I ... started using a reset script for the PCI which is required with some GPU

This is more or less what is required, in case you want to attempt 2 GPU in your machine

1. erased all configuration added to the boot loader (mostly due to updates to proxmox)
   1. only option left is nomodeset
   2. fun things is boot parameters actually permit to output the GPU (thus VM) to a directly connected monitor
2. prevented any and all GPU related modules from loading at proxmox
3. set-up just one VM with pass-through enabled PCI
   1. set the correct PCI number under 'raw device'
   2. enable 'Primary GPU'
   3. [ advanced ] enable 'rom bar'
   4. [ advanced ] enable PCI-express
4. boot and check if the VM shows the pci device
   1. lspci
5. take a snapshot of the VM if it does show up
6. take time to test and try the right GPU driver
   1. test and snapshot again if all works well

Let us know if this is useful in case you try the GPU again

These modules are loaded by proxmox and to my belief required for pass-through to work well

vfio_pci
vfio_pci_core
irqbypass
vfio_iommu_type1
vfio
iommufd