[SOLVED] This device cannot start. (Code 10)

[DataOk0101]

I'm passing through an Nvidia A2 on a Dell R250
I've read these directions: https://pve.proxmox.com/wiki/PCI_Passthrough
My device shows up in Windows 2022, but after I load the driver, I get

This device cannot start. (Code 10)

Insufficient system resources exist to complete the API.

I modified my grub line as follows:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifbff"

Afterwards, I found this error:

Mar 20 21:54:27 sfmi-pve01 QEMU[1610]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 20 21:54:27 sfmi-pve01 QEMU[1610]: kvm: vfio_region_read(0000:01:00.0:region0+0x0, 4) failed: Device or resource busy
Mar 20 21:54:27 sfmi-pve01 QEMU[1610]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 20 21:54:27 sfmi-pve01 QEMU[1610]: kvm: vfio_region_read(0000:01:00.0:region0+0x4, 4) failed: Device or resource busy
Mar 20 21:54:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 20 21:54:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 20 21:54:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 20 21:54:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]

I still can't start the device in Windows; any suggestions?

 

[leesteken]

I modified my grub line as follows:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifbff"

Afterwards, I found this error:

Mar 20 21:54:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]

video=efifb:off does nothing anymore. When doing passthrough of the boot GPU, you need this two-year old work-around: https://forum.proxmox.com/posts/478351/ That typically solves that error message you are getting.

 

[DataOk0101]

I removed video-efifbff and added initcall_blacklist=sysfb_init, still getting the same code 10 in windows and the following in syslog:

Mar 21 08:23:07 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 08:23:07 sfmi-pve01 pvedaemon[1196]: <root@pam> end task UPID:sfmi-pve01:00000F60:0001CCD9:65FC26A9:qmstart:103:root@pam: OK
Mar 21 08:23:15 sfmi-pve01 QEMU[3980]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 21 08:23:15 sfmi-pve01 QEMU[3980]: kvm: vfio_region_read(0000:01:00.0:region0+0x0, 4) failed: Device or resource busy
Mar 21 08:23:15 sfmi-pve01 QEMU[3980]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 21 08:23:15 sfmi-pve01 QEMU[3980]: kvm: vfio_region_read(0000:01:00.0:region0+0x4, 4) failed: Device or resource busy
Mar 21 08:23:15 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 08:23:15 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 08:23:15 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 08:23:15 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]

 

[leesteken]

I removed video-efifbff and added initcall_blacklist=sysfb_init, still getting the same code 10 in windows and the following in syslog:

What is the output of cat /proc/cmdline? Do you early bind the GPU to vfio-pci (to make sure no drives touched the GPU before the VM starts): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_host_configuration ?

 

[DataOk0101]

Yes, I have also blacklisted nvidiafb, nouveau

root@sfmi-pve01:/etc/modprobe.d# lspci -k | grep -A 3 "A2"
01:00.0 3D controller: NVIDIA Corporation GA107GL [A2 / A16] (rev a1)
Subsystem: NVIDIA Corporation GA107GL [A2 / A16]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID 12GSAS/PCIe Secure SAS39xx
root@sfmi-pve01:/etc/modprobe.d# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.13-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init
root@sfmi-pve01:/etc/modprobe.d#

What I'm seeing here is "Enabling HDA controller", which I do not believe an Nvida A2 has a high-definition audio controller.

Mar 21 20:14:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: Enabling HDA controller
Mar 21 20:14:27 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 20:14:27 sfmi-pve01 pvedaemon[1200]: <root@pam> end task UPID:sfmi-pve01:000015B0:0002CB7B:65FCCD61:qmstart:103:root@pam: OK
Mar 21 20:14:35 sfmi-pve01 QEMU[5574]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 21 20:14:35 sfmi-pve01 QEMU[5574]: kvm: vfio_region_read(0000:01:00.0:region0+0x0, 4) failed: Device or resource busy
Mar 21 20:14:35 sfmi-pve01 QEMU[5574]: kvm: vfio_region_read(0000:01:00.0:region0+0xa00, 4) failed: Device or resource busy
Mar 21 20:14:35 sfmi-pve01 QEMU[5574]: kvm: vfio_region_read(0000:01:00.0:region0+0x4, 4) failed: Device or resource busy
Mar 21 20:14:35 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 20:14:35 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 20:14:35 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]
Mar 21 20:14:35 sfmi-pve01 kernel: vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0x81000000-0x81ffffff]

 

[DataOk0101]

What is the output of cat /proc/cmdline? Do you early bind the GPU to vfio-pci (to make sure no drives touched the GPU before the VM starts): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_host_configuration ?

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:25b6 disable_vga=1

 

[DataOk0101]

Update, this fixed it:
https://support.hpe.com/hpesc/public/docDisplay?docId=a00112218en_us&docLocale=en_US