GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)

[WeirdGoesPro]

I am having issues passing through my new GPU to my Windows 11 VM. I previously had another AMD GPU hooked up to this same VM without issue. The new 9070 GPU will display the Proxmox terminal just fine, so I’m relatively sure there is no defect with the device.

CPU: AMD Ryzen 9 5900X
GPU: AMD RX 9070 XT
MB: NZXT N7 B550
PWR: 850 Watt

IOMMU: Enabled
Above 4G Decoding: Enabled
Re-Size Bar Support: Enabled

When I try to start the VM, I get this error:

error writing ‘1’ to ‘/sys/bus/pci/devices/0000:08:00.0/reset’: Inappropriate ioctl for device
failed to reset PCI device ‘0000:08:00.0’, but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: vfio: Unable to power on device, stuck in D3
kvm: vfio: Unable to power on device, stuck in D3
TASK OK

Device:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [RX 9070/9070 XT] [1002:7550] (rev c0) (prog-if 00 [VGA controller])
Subsystem: XFX Limited Device [1eae:8811]

/etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=‘lsb_release -i -s 2> /dev/null || echo Debian’
GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_iommu=on iommu=pt rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1”
GRUB_CMDLINE_LINUX=“”

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/vfio.conf:

options vfio-pci ids=0000:08:00.0,0000:08:00.1
options kvm ignore_msrs=1
options vfio-pci ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1
softdep radeon pre: vfio vfio_pci

VM.conf: I downloaded and added the ROM to the VM configuration file.

hostpci0: 0000:08:00,pcie=1,x-vga=1,romfile=9070xt.rom

At this point I feel totally lost in the weeds, and would greatly appreciate any help that could point me in the right direction to fix this. Thanks!

UPDATE: the error message changed somewhere between updating the bios and removing the rom entry from the vm.conf.

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: ../hw/pci/pci.c:1654: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
stopping swtpm instance (pid 10043) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

 

[the1corrupted]

Can you verify the VFIO driver is loading for the GPU?

Run lspci -nnk and make sure that vfio-pci is the driver being used on the host.

The error is reporting the device is stuck in a D3 state which you have disabled. Update grub, initram and reboot the host.

The romfile should be optional, try removing it. I think with rebar there is a 'romrebar=X' argument where X is an integer.

 

[WeirdGoesPro]

Can you verify the VFIO driver is loading for the GPU?

Run lspci -nnk and make sure that vfio-pci is the driver being used on the host.

The error is reporting the device is stuck in a D3 state which you have disabled. Update grub, initram and reboot the host.

The romfile should be optional, try removing it. I think with rebar there is a 'romrebar=X' argument where X is an integer.

lspci revealed the following:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [RX 9070/9070 XT] [1002:7550] (rev c0)
Subsystem: XFX Limited Device [1eae:8811]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu

If I'm reading that right, it looks like vfio-pci is being used for the GPU. Grub and initram have been updated any time a change was made to those files, and a reboot was done each time. I've removed the rom entry from the conf file too.

No effect, I'm still getting the D3 error.

Edit: the error message changed somewhere between updating the bios and removing the rom entry from the vm.conf.

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: ../hw/pci/pci.c:1654: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
stopping swtpm instance (pid 10043) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

 

[Chris123456]

Hi!
I have an similar error.
But my GPU stuck in this error only after rebooting the VM.
Rebooting the host will start the GPU without a Problem.

Can you also reproduce this behaviour?

Regards
Chris

 

[WeirdGoesPro]

Hi!
I have an similar error.
But my GPU stuck in this error only after rebooting the VM.
Rebooting the host will start the GPU without a Problem.

Can you also reproduce this behaviour?

Regards
Chris

Unfortunately no, rebooting the machine has no effect.