[SOLVED] Quadro GPU Passthrough to Windows shows code 43

[mega_neo]

Hi Everybody,

Unfortunately I'm starting one more post regarding GPU passthrough, like many before. I've been reading several different posts and trying some different options but to no avail. I'm even trying the new nvidia drivers that are said to allow GPU passthrough but that also didn't help.

I thought my life would be easier since I'm working with a nvidia Quadro GPU but that doesn't seem to be the truth.
I hope you can get me in the right direction so I can finally solve this stupid error 43.

---

So, I'm using a HP workstation:

Machine: HP Z230 SFF
CPU: i7 4790
Ram: 8GB (to be updated to 32 when i find the 4 ram sticks)
GPU: Nvidia Quadro K1200
SDD: 120 GB SSD for Proxmox (with the VM disk inside it-also to be updated).

Using Proxmox VE 6.4-5
Guest VM: Windows 10 Pro

Following the several manuals around, I started by adding the Grub options to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream video=vesafb:off,efifb:off"

I'm using the ACS option due to the fact that my GPU was in the same IOMMU group as the PCI bridge PCIe x16 controller. I also tried the multifunction option put didn't se a major difference.
*EDIT*: I was told that the grouping with the bridge is not a problem since the OS is able to work it that, so I took the ACS option out. Just wanted to let this here to show what I've tried already.

I've also added the required modules to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Blacklisted the drivers in /etc/modprobe.d/blacklist.conf and in /etc/modprobe.d/pve-blacklist.conf as I've seen different manuals and didn't want to be missing it:

blacklist nvidiafb
blacklist nvidia
blacklist radeon
blacklist nouveau

Added the GPU IDs to /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:13bc,10de:0fbc disable_vga=1

and have this as my VM config:

agent: 1
args: -machine max-ram-below-4g=1G,kernel_irqchip=on
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-100-disk-1,size=4M
hostpci0: 01:00,pcie=1,romfile=vbios_nvk1200.bin,x-vga=1
machine: pc-q35-5.2
memory: 4096
name: W10
net0: virtio=7E:45:B1:6B:A3:1F,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: After_install
scsi0: local-lvm:vm-100-disk-0,cache=writeback,discard=on,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=848c8dce-ffd2-4db4-822a-6e80249123e7
sockets: 1
vmgenid: 2ab791dc-6c9c-4420-be74-b259eaa8a9e8

---

So, that's it. I've been fighting with this for a couple of weeks without a solution in sight.
I see the GPU being recognized in Windows so I think the VM on itself might not be the problem. I'm even able to install the most up to date driver.
If the problem was a recognition i think the driver would not even install as it doesn't see a nvidia GPU.

So I really don't know where to look any more tips for this. I'm really out of resources.
If you guys could have a look at the configs and see if you spot anything wrong or something missing please let me know.

Thank you in advance

<I also posted this request on reddit r/proxmox first to check if the help would be faster but also with no result>

 

[mega_neo]

I think I got it.

In my case, I found a website talking about what changed in Proxmox VE 6.2 and how to fix the GPU passthrough.

From Proxmox 6.2 the module

vfio_iommu_type1

is already included in the kernel, so that may change somethings.

I follow the instructions on they solved their issue, and adding:

vfio_iommu_type1.allow_unsafe_interrupts=1

in the grub file, also solved my issue.

Don't have code 43 in the device manager and GPU-z shows not only the nvidia details (as in the past), but also the sensors are on while before only had to entries with no changes.

Maybe this can helps anyone in the future.


There was also another change I did. I took out the romfile fro my VM config since, as indicated by someone else on my reddit post, the romfile i extracted maybe was not having the correct template or even not correct at all due to:

If your host is initialising the card on startup (e.g. it's set as the primary GPU) this dumps the vBIOS that already got trashed by initialisation, so it's worthless. You get precisely the same result by not using the romfile argument at all.

On the other hand, if your host isn't initialising the card on startup, you don't need to supply a clean vBIOS file anyway, since it hasn't been ruined by initialisation.

This method is only useful for systems where you can temporarily add another GPU as the primary, leaving your target GPU unmolested. Then you can dump the clean vBIOS, remove the temporary GPU, and you're good to go.

by thenickdude

 

[LocalUser]

Hi,

I'm using an HP640 Workstation with,
CPU: Intel Xeon E5-2650 v4
RAM: 32GB
GPU: Quadro K2200

basically (or at least close to) your system @mega_neo. However, I can't seem to get the past "Code 43".

/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream video=efifb:off,vesafb:off"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_OS_PROBER=true

/etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

/etc/modprobe.d/pve-blacklist.conf

blacklist nvidiafb

/etc/modprobe.d/vfio.conf

# options vfio-pci ids=10de:13ba,10de:0fbc disable_vga=1

VM config:

args: -machine max-ram-below-4g=1G,kernel_irqchip=on -cpu host,kvm=off
balloon: 0
boot: order=ide0;ide2;net0
cores: 8
cpu: host,hv-vendor-id=HPZ640
hostpci0: 02:00,pcie=1,x-vga=1
ide0: VMs1:vm-100-disk-0,size=32G
ide2:
machine: pc-q35-5.2
memory: 16384
name: Win10
net0: e1000=96:8A:D1:D6:05:87,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=0af6bb74-4223-4e12-9672-3bae9cece8cb
sockets: 1
unused0: VMs1:vm-100-disk-1
vmgenid: 57f3f16f-5192-410a-a9e1-d74647adedf7

update-grub
update-initramfs -u
reboot


Adding "pc" to args: -machine as stated in the tutorial yields the following error.
kvm:/usr/share/qemu-server/pve-q35-4.0.cfg:1: Bus 'pcie.0' not found
TASK ERROR: start failed: QEMU exited with code 1

removing pc makes it bootable, with "Code 43".

Without commenting out vfio.conf the following error arises,
kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio 0000:02:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

Since you have a consumer CPU you have the iGPU enabled for the host I guess? Is that where it falls apart for me...

 

[LocalUser]

Solved mine (Code 43) with a nudge from @avw by adjusting,

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream video=efifbff video=vesafbff"

/etc/modprobe.d/vfio.conf <- removed the # thus enabling it again
options vfio-pci ids=10de:13ba,10de:0fbc disable_vga=1

VM config:

Spoiler

args: -machine max-ram-below-4g=1G,kernel_irqchip=on
balloon: 0
boot: order=ide0;ide2;net0
cores: 4
hostpci0: 02:00,pcie=1
ide0: VMs1:vm-100-disk-0,size=32G
ide2: local:iso/en_windows_10_consumer_editions_version_20h2_updated_nov_2020_x$
machine: pc-q35-5.2
memory: 16384
name: Win10
net0: e1000=96:8A16:05:87,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata1: WD_Green:vm-100-disk-0,size=250G
scsihw: virtio-scsi-pci
smbios1: uuid=0af6bb74-4223-4e12-9672-3bae9cece8cb
sockets: 1
unused0: VMs1:vm-100-disk-1
vga: none

Thus, not hiding the fact that it's a VM seems to work out just fine even though there is only 1 GPU present in the system.

 

[leesteken]

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream [U]video=efifb:off video=vesafb:off"[/U]

/etc/modprobe.d/vfio.conf <- removed the # thus enabling it again
options vfio-pci ids=10de:13ba,10de:0fbc disable_vga=1

...

Thus, not hiding the fact that it's a VM seems to work out just fine even though there is only 1 GPU present in the system.

Hiding used to be necessary for NVidia consumer cards because their drivers did not support runnng in VMs. This is not related to having 1 GPU nor applicable to Quadro's.
The main fix was splitting the single video= parameter to two separate ones (because the video parameter does not support multiple values separate by commas).
Just curious: Do you really need pcie_acs_override? I would advice against it because it break proper isolation. Do you also need the vfio-pci ids? Does this not prevent any display of stating Proxmox and make troubleshooting much harder? I would expect a Quadro to properly reset without binding it to vfio-pci manually.

 

[LocalUser]

Hiding used to be necessary for NVidia consumer cards because their drivers did not support runnng in VMs. This is not related to having 1 GPU nor applicable to Quadro's.
The main fix was splitting the single video= parameter to two separate ones (because the video parameter does not support multiple values separate by commas).
Just curious: Do you really need pcie_acs_override? I would advice against it because it break proper isolation. Do you also need the vfio-pci ids? Does this not prevent any display of stating Proxmox and make troubleshooting much harder? I would expect a Quadro to properly reset without binding it to vfio-pci manually.

The main fix was indeed splitting the videoparameter.

The pcie_acs_override parameter is not necessary. Removed it from grub config and rebooted, GPU in the VM still works.

vfio-pci ids is not necessary if you want to connect a screen to the GPU. However, now it is impossible to connect by Remote Desktop into the VM with GPU passed to it. Why it behaves like that now is unclear to me. Must be that the host is accessing the GPU now for some reason after trying with/without vfio-pci ids. Could that be it... Error in the VM is now Code 12. Does it make sense?

I am running Proxmox headless, and I would like to have remote access to some VMs with the ability to use the GPU.

Edit:
The VM is unable to start with the passed GPU as primary (x-vga=1)
Procedure Removed the passed GPU from the VM, rebooted, then RDP was working again.
Procedure Added the passed GPU to the VM again, rebooted, then RDP was working with Code 12 on the GPU (maybe drivers).
Procedure Rebooted the VM again, then RDP was not working. However, the GPU outputs the VM on a screen so that is working.

 

[LocalUser]

Solution to my problem:

In the Windows VM Edit: [Local Group Policy Editor->Local Computer Policy->Administrative Templates->Windows Components->Remote Desktop Services->Remote Desktop Session Host->Remote Session Enviroment], set the Policy [Use WDDM graphics display driver for Remote Desktop Connections] to Disabled.