GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)

W

WeirdGoesPro

New Member
May 30, 2025
3
0
1
I am having issues passing through my new GPU to my Windows 11 VM. I previously had another AMD GPU hooked up to this same VM without issue. The new 9070 GPU will display the Proxmox terminal just fine, so I’m relatively sure there is no defect with the device.

CPU: AMD Ryzen 9 5900X
GPU: AMD RX 9070 XT
MB: NZXT N7 B550
PWR: 850 Watt

IOMMU: Enabled
Above 4G Decoding: Enabled
Re-Size Bar Support: Enabled

When I try to start the VM, I get this error:
error writing ‘1’ to ‘/sys/bus/pci/devices/0000:08:00.0/reset’: Inappropriate ioctl for device
failed to reset PCI device ‘0000:08:00.0’, but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: vfio: Unable to power on device, stuck in D3
kvm: vfio: Unable to power on device, stuck in D3
TASK OK
Click to expand...

Device:
08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [RX 9070/9070 XT] [1002:7550] (rev c0) (prog-if 00 [VGA controller])
Subsystem: XFX Limited Device [1eae:8811]
Click to expand...

/etc/default/grub:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=‘lsb_release -i -s 2> /dev/null || echo Debian’
GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_iommu=on iommu=pt rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1”
GRUB_CMDLINE_LINUX=“”
Click to expand...

/etc/modules:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Click to expand...

/etc/modprobe.d/vfio.conf:
options vfio-pci ids=0000:08:00.0,0000:08:00.1
options kvm ignore_msrs=1
options vfio-pci ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1
softdep radeon pre: vfio vfio_pci
Click to expand...

VM.conf: I downloaded and added the ROM to the VM configuration file.
hostpci0: 0000:08:00,pcie=1,x-vga=1,romfile=9070xt.rom
Click to expand...

At this point I feel totally lost in the weeds, and would greatly appreciate any help that could point me in the right direction to fix this. Thanks!

UPDATE: the error message changed somewhere between updating the bios and removing the rom entry from the vm.conf.

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: ../hw/pci/pci.c:1654: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
stopping swtpm instance (pid 10043) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1
Click to expand...
 
Last edited:
Can you verify the VFIO driver is loading for the GPU?

Run lspci -nnk and make sure that vfio-pci is the driver being used on the host.

The error is reporting the device is stuck in a D3 state which you have disabled. Update grub, initram and reboot the host.

The romfile should be optional, try removing it. I think with rebar there is a 'romrebar=X' argument where X is an integer.
 
Last edited:
the1corrupted said:
Can you verify the VFIO driver is loading for the GPU?

Run lspci -nnk and make sure that vfio-pci is the driver being used on the host.

The error is reporting the device is stuck in a D3 state which you have disabled. Update grub, initram and reboot the host.

The romfile should be optional, try removing it. I think with rebar there is a 'romrebar=X' argument where X is an integer.
Click to expand...

lspci revealed the following:
08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [RX 9070/9070 XT] [1002:7550] (rev c0)
Subsystem: XFX Limited Device [1eae:8811]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
Click to expand...

If I'm reading that right, it looks like vfio-pci is being used for the GPU. Grub and initram have been updated any time a change was made to those files, and a reboot was done each time. I've removed the rom entry from the conf file too.

No effect, I'm still getting the D3 error.

Edit: the error message changed somewhere between updating the bios and removing the rom entry from the vm.conf.

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: ../hw/pci/pci.c:1654: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
stopping swtpm instance (pid 10043) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1
Click to expand...
 
Last edited:
Hi!
I have an similar error.
But my GPU stuck in this error only after rebooting the VM.
Rebooting the host will start the GPU without a Problem.

Can you also reproduce this behaviour?

Regards
Chris
 
Chris123456 said:
Hi!
I have an similar error.
But my GPU stuck in this error only after rebooting the VM.
Rebooting the host will start the GPU without a Problem.

Can you also reproduce this behaviour?

Regards
Chris
Click to expand...
Unfortunately no, rebooting the machine has no effect.
 
WeirdGoesPro said:
I am having issues passing through my new GPU to my Windows 11 VM. I previously had another AMD GPU hooked up to this same VM without issue. The new 9070 GPU will display the Proxmox terminal just fine, so I’m relatively sure there is no defect with the device.

CPU: AMD Ryzen 9 5900X
GPU: AMD RX 9070 XT
MB: NZXT N7 B550
PWR: 850 Watt
Click to expand...

I think i have the same problem but i have created new thread.
Were you be able to fix this ? Already fighting with that 2 weeks and i need to fix this asap ! Got issue with RTX5090 and RTX PRO 6000 so i guess this might be passthrough issue rather than nvidia/amd specific.
My thread
 
Regarding Radeon RX 9070 XT: after 2 Days of try and error i was able to passtough the GPU. Running stable now for 2 days.

The secret sauce was to use the newest Kernel ( Linux 6.14.8-2-bpo12-pve) which you have to install separatly and not to bind the card to vfio. I dont know exactly whats going on, but after the card is used by kernel-module "amdgpu" only (not vfio!) I was able to see/use the card in vm-guests like Bazzite (Linux) and Windows 11. For Windows it is necessary to use the romfile for your card. Mine is ASUS Radeon Prime 9070 XT OC.

Unfortunatly even the AMD 90xx-Series has the well known reset-bug which you have to deal with.
 
I have been using it for several months with the following settings and have not encountered any problems.

Due to a bug in vfio, the latest kernel is required.

It is not necessary to specify vBIOS. Perhaps it is not a ROM that should be specified.
Specifying disable_idle_d3=1 will cause the problem to worsen.

There will be no problems even if you restart or shut down the virtual machine.
When the virtual machine is stopped, amdgpu is used, but when it is started, vfio-pci is used.

・intel (Please do not include unnecessary options)
sed -i '1s/$/ quiet iommu=pt split_lock_detect=off/g' /etc/kernel/cmdline
sed -i '/GRUB_CMDLINE_LINUX_DEFAULT=/c GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt split_lock_detect=off"' /etc/default/grub

・AMD (Probably only this setting is necessary)
sed -i '1s/$/ quiet iommu=pt/g' /etc/kernel/cmdline
sed -i '/GRUB_CMDLINE_LINUX_DEFAULT=/c GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"' /etc/default/grub

cat << EOF > /etc/modules
vfio
vfio_iommu_type1
vfio_pci
EOF
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf

rx9070xt PCI pass through

mkdir -p /var/lib/vz/snippets
nano /var/lib/vz/snippets/rx9070_reset.sh
---
#!/bin/bash

phase="$2"
echo "Phase is $phase"
if [ "$phase" == "pre-start" ]; then
# Unbind gpu from amdgpu
echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null
sleep 2
# Resize the GPU's BAR2 memory region (useful for PCI passthrough)
echo 8 > /sys/bus/pci/devices/0000:04:00.0/resource2_resize
sleep 2
elif [ "$phase" == "post-stop" ]; then
# Unbind gpu from vfio-pci
sleep 5
echo "0000:04:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
sleep 2
# Bind amdgpu
echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
sleep 2
fi
---
chmod +x /var/lib/vz/snippets/rx9070_reset.sh

cat /var/lib/vz/snippets/rx9070_reset.sh

qm set 923 -args '-cpu host,hv_passthrough,-hypervisor,level=35,+vmx'
qm set 923 -hookscript local:snippets/rx9070_reset.sh
qm set 923 -hostpci0 0000:04:00,pcie=1,rombar=0,x-vga=1

qm showcmd 923 --pretty true

update-initramfs -u -k all
proxmox-boot-tool refresh
update-grub
reboot
Click to expand...

Edited: Removed kernel option intel_iommu=on. No longer needed in kernel 6.8 and later. Kernels prior to 6.8 are at end-of-life.
 
Last edited:
  • Like
Reactions: 9qwrog1y
Thx for the script.
Does this also apply for NVIDIA GPUs?

Since there is the VM ID inside the script I assume we need to do this for each GPU which is passed through?

Best regards
Chris
 
I don't think this is what Nvidia's general procedure entails.

I have only confirmed this procedure with rx9070 or rx9070xt.
 
Hey, thank you for the commands and the snippet. This was kinda driving me crazy, I can confirm this worked for me on my RX 9070XT.
 
  • Like
Reactions: uzumo
You must log in or register to reply here.
Top Bottom