VM GPU Passthrough RTX 5060 TI

X

xragnaz

New Member
Jul 1, 2025
3
0
1
Hello, I'm having a problem with GPU passthrough in my VM. Any help would be appreciated.

I was able to get passthrough working on an LXC as an alternative, but the fact that it doesn't work on a VM is giving me a headache.

When I try to start the VM, it always gives the same error:
Code: 
error writing '1' to '/sys/bus/pci/devices/0000:01:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:01:00.0', but trying to continue as not all devices need a reset
kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0: vfio 0000:01:00.0: error getting device from group 9: No such device
Verify all devices in group 9 are bound to vfio-<bus> or pci-stub and not already in use
TASK ERROR: start failed: QEMU exited with code 1

PVE Version:
Code: 
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)

I think I followed all the correct steps and tried different configurations:

1) /etc/default/grub:
Code: 
 GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

2) lspci -nnk
Code: 
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2d04] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:41af]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22eb] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:0000]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

3) dmesg | grep -e DMAR -e IOMMU
Code: 
[    0.008591] ACPI: DMAR 0x0000000041C49000 000098 (v01 ALASKA A M I    01072009 AMI  01000013)
[    0.008639] ACPI: Reserving DMAR table memory at [mem 0x41c49000-0x41c49097]
[    0.046923] DMAR: IOMMU enabled
[    0.119557] DMAR: Host address width 42
[    0.119558] DMAR: DRHD base: 0x000000fc800000 flags: 0x0
[    0.119573] DMAR: dmar0: reg_base_addr fc800000 ver 7:0 cap e9de008cee690462 ecap 1aca9a00f0ef5e
[    0.119595] DMAR: DRHD base: 0x000000fc810000 flags: 0x1
[    0.119605] DMAR: dmar1: reg_base_addr fc810000 ver 7:0 cap e9de008cee690462 ecap 1aca9a00f0efde
[    0.119624] DMAR: SATC flags: 0x1
[    0.119627] DMAR-IR: IOAPIC id 2 under DRHD base  0xfc810000 IOMMU 1
[    0.119628] DMAR-IR: HPET id 0 under DRHD base 0xfc810000
[    0.119629] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.121197] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.471508] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.538958] DMAR: No RMRR found
[    0.538959] DMAR: No ATSR found
[    0.538960] DMAR: IOMMU feature sc_support inconsistent
[    0.538962] DMAR: dmar0: Using Queued invalidation
[    0.538965] DMAR: dmar1: Using Queued invalidation
[    0.543856] DMAR: Intel(R) Virtualization Technology for Directed I/O

4) /etc/modprobe.d/vfio.conf:
Code: 
options vfio-pci ids=10de:2d04,10de:22eb disable_vga=1

5) /etc/modprobe.d/pve-blacklist.conf
Code: 
blacklist nvidiafb
blacklist nouveau
blacklist snd_hda_intel
blacklist nvidia

6) IOMMU Groups:
Code: 
IOMMU Group /sys/kernel/iommu_groups/0/devices 00:02.0 VGA compatible controller [0300]: Intel Corporation Arrow Lake-S [Intel Graphics] [8086:7d67] (rev 06)
IOMMU Group /sys/kernel/iommu_groups/10/devices 80:14.0 USB controller [0c03]: Intel Corporation Device [8086:7f6e] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/11/devices 80:14.5 Non-VGA unclassified device [0000]: Intel Corporation Device [8086:7f2f] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/12/devices 80:15.0 Serial bus controller [0c80]: Intel Corporation Device [8086:7f4c] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/12/devices 80:15.1 Serial bus controller [0c80]: Intel Corporation Device [8086:7f4d] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/12/devices 80:15.2 Serial bus controller [0c80]: Intel Corporation Device [8086:7f4e] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/12/devices 80:15.3 Serial bus controller [0c80]: Intel Corporation Device [8086:7f4f] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/13/devices 80:16.0 Communication controller [0780]: Intel Corporation Device [8086:7f68] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/14/devices 80:17.0 SATA controller [0106]: Intel Corporation Device [8086:7f62] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/15/devices 80:19.0 Serial bus controller [0c80]: Intel Corporation Device [8086:7f7a] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/15/devices 80:19.1 Serial bus controller [0c80]: Intel Corporation Device [8086:7f7b] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/16/devices 80:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7f44] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/17/devices 80:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:7f36] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/18/devices 80:1e.0 Communication controller [0780]: Intel Corporation Device [8086:7f28] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/18/devices 80:1e.3 Serial bus controller [0c80]: Intel Corporation Device [8086:7f2b] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/19/devices 80:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7f06] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/19/devices 80:1f.3 Audio device [0403]: Intel Corporation Device [8086:7f50] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/19/devices 80:1f.4 SMBus [0c05]: Intel Corporation Device [8086:7f23] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/19/devices 80:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:7f24] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/1/devices 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:7d1b] (rev 01)
IOMMU Group /sys/kernel/iommu_groups/20/devices 81:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV3 NVMe SSD SM2268XT2 (DRAM-less) [2646:5028] (rev 03)
IOMMU Group /sys/kernel/iommu_groups/21/devices 82:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group /sys/kernel/iommu_groups/2/devices 00:04.0 Signal processing controller [1180]: Intel Corporation Device [8086:ad03] (rev 01)
IOMMU Group /sys/kernel/iommu_groups/3/devices 00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:ae4d] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/4/devices 00:08.0 System peripheral [0880]: Intel Corporation Device [8086:ae4c] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/5/devices 00:0a.0 Signal processing controller [1180]: Intel Corporation Device [8086:ad0d] (rev 01)
IOMMU Group /sys/kernel/iommu_groups/6/devices 00:0b.0 Processing accelerators [1200]: Intel Corporation Arrow Lake NPU [8086:ad1d] (rev 01)
IOMMU Group /sys/kernel/iommu_groups/7/devices 00:14.0 RAM memory [0500]: Intel Corporation Device [8086:ae7f] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/8/devices 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:ae0d] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/8/devices 00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:ae23] (rev 10)
IOMMU Group /sys/kernel/iommu_groups/9/devices 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2d04] (rev a1)
IOMMU Group /sys/kernel/iommu_groups/9/devices 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22eb] (rev a1)

7) update-initramfs -u -k all && update-grub && reboot

8) VM config

1751394905854.png1751394977469.png
 
Last edited:
not sure it's related to the issue, but your screenshot shows that rom-bar is checked, but the error message indicates that it's unchecked ",rombar=0" did you try both?

the full vmconfig and the output of dmesg would be interesting:

Code: 
qm config ID
dmesg
 
dcsapak said:
not sure it's related to the issue, but your screenshot shows that rom-bar is checked, but the error message indicates that it's unchecked ",rombar=0" did you try both?

the full vmconfig and the output of dmesg would be interesting:

Code: 
qm config ID
dmesg
Click to expand...
Yes, the error message was from another attempt, but the error was always the same.

Code: 
bios: ovmf
boot: order=sata0;ide2;net0
cores: 1
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,x-vga=1
ide2: local:iso/debian-12.11.0-amd64-DVD-1.iso,media=cdrom,size=3760M
machine: q35
memory: 2048
meta: creation-qemu=9.2.0,ctime=1751392650
name: test
net0: virtio=BC:24:11:C1:F7:E2,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
sata0: local-lvm:vm-100-disk-1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=f27ac9c5-a1e5-4f83-bae0-1cd431dc19b8
sockets: 1
vmgenid: 27ccbf82-6d88-4cc2-a719-6f8b70b42bfa

dmesg output is attached.

Thank you.
 

Attachments

Last edited:
from the dmesg output it looks like the nvidia driver is loaded, can you try to blacklist that and reboot? after a reboot, before the vm start 'lspci -nnk' should say 'vfio-pci' for the used driver
(you can also post the 'lspci -nnk' output here if you want)
 
dcsapak said:
from the dmesg output it looks like the nvidia driver is loaded, can you try to blacklist that and reboot? after a reboot, before the vm start 'lspci -nnk' should say 'vfio-pci' for the used driver
(you can also post the 'lspci -nnk' output here if you want)
Click to expand...
Hi! I already posted the output of lspci -nnk in the original post and there is my blacklist file too. Right now, I'm using passthrough with an LXC because the client needed it, but when I tried passthrough with the VM, the driver shows as "vfio-pci". Thank you.
 
Hello,

(probably) the same problem with 5060 TI here. Tried kernel version 6.8.4, 6.8.12 and 6.14.5, since people were reporting kernel version related problems. No success so far. dmesg shows:


Code: 
[   74.464956] vfio-pci 0000:01:00.0: vgaarb: deactivate vga console
[   74.464959] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   76.006719] pcieport 0000:00:01.0: broken device, retraining non-functional downstream link at 2.5GT/s
[   77.007683] pcieport 0000:00:01.0: retraining failed
[   77.007694] pcieport 0000:00:01.0: Data Link Layer Link Active not set in 1000 msec
[   77.007718] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.067900] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.067905] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068051] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   77.068322] vfio-pci 0000:01:00.0: vgaarb: deactivate vga console
[   77.068324] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   77.068326] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068331] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068689] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068698] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.068701] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.068704] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.068733] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068910] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.068916] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.068921] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.068924] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.069218] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.069246] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   77.069368] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.069497] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   77.069502] vfio-pci 0000:01:00.0: resetting
[   77.069503] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible

Or with kernel 6.8.4:

Code: 
[    1.967900] vfio-pci 0000:01:00.0: vgaarb: deactivate vga console
[    1.967907] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[    1.968016] vfio_pci: add [10de:2d04[ffffffff:ffffffff]] class 0x000000/00000000
[    6.764789] vfio-pci 0000:01:00.0: not ready 1023ms after resume; waiting
[    7.810787] vfio-pci 0000:01:00.0: not ready 2047ms after resume; waiting
[    9.922809] vfio-pci 0000:01:00.0: not ready 4095ms after resume; waiting
[   14.466787] vfio-pci 0000:01:00.0: not ready 8191ms after resume; waiting
[   23.170794] vfio-pci 0000:01:00.0: not ready 16383ms after resume; waiting
[   40.066790] vfio-pci 0000:01:00.0: not ready 32767ms after resume; waiting
[   74.882790] vfio-pci 0000:01:00.0: not ready 65535ms after resume; giving up
[   74.882821] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[   74.942869] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   75.002885] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   75.002910] vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible
[   75.003018] vfio_pci: add [10de:22eb[ffffffff:ffffffff]] class 0x000000/00000000
[   75.003320] vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible


Is it possible that the RTX 5060 TI just isn't passthrough-able?

Kind regards,
Tom
 
You must log in or register to reply here.
Top Bottom