Warning: UPDATING these packages broke my pci passthrough.

[v95klima]

Just broke when updated yesterday. All VMs depending on AMD 6900 XT stopped working. Was working for last 4 years, and now with update broke.
Please help?

error writing '1' to '/sys/bus/pci/devices/0000:04:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:04:00.0', but trying to continue as not all devices need a reset
kvm: -device vfio-pci,host=0000:01:00.2,id=hostpci2,bus=ich9-pcie-port-3,addr=0x0: vfio 0000:01:00.2: error getting device info: Input/output error
TASK ERROR: start failed: QEMU exited with code 1

dmesg | grep vfio-pci
[ 5.355747] vfio-pci 0000:11:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 236.763142] vfio-pci 0000:04:00.0: vgaarb: deactivate vga console
[ 236.763152] vfio-pci 0000:04:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 236.763448] vfio-pci 0000:04:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=io+memwns=none
[ 236.763552] vfio-pci 0000:04:00.0: vgaarb: deactivate vga console
[ 236.763556] vfio-pci 0000:04:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 239.572122] vfio-pci 0000:04:00.0: enabling device (0002 -> 0003)
[ 239.573632] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)
[ 239.577215] vfio-pci 0000:01:00.2: enabling device (0000 -> 0002)
[ 239.683537] vfio-pci 0000:01:00.2: vfio: runtime resume failed -13

cat /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifbff amd_pstate=passive"
GRUB_CMDLINE_LINUX=""


root@pve:~# pveversion

pve-manager/8.4.5/57892e8e686cb35b (running kernel: 6.8.12-12-pve)

root@pve:~# pveversion -v

proxmox-ve: 8.4.0 (running kernel: 6.8.12-12-pve)

pve-manager: 8.4.5 (running version: 8.4.5/57892e8e686cb35b)

proxmox-kernel-helper: 8.1.1

proxmox-kernel-6.8.12-12-pve-signed: 6.8.12-12

proxmox-kernel-6.8: 6.8.12-12

proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11

proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10

proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6

proxmox-kernel-6.5: 6.5.13-6

proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4

amd64-microcode: 3.20240820.1~deb12u1

ceph-fuse: 17.2.7-pve1

corosync: 3.1.9-pve1

criu: 3.17.1-2+deb12u1

glusterfs-client: 10.3-5

ifupdown2: 3.2.0-1+pmx11

ksm-control-daemon: 1.5-1

libjs-extjs: 7.0.0-5

libknet1: 1.30-pve2

libproxmox-acme-perl: 1.6.0

libproxmox-backup-qemu0: 1.5.2

libproxmox-rs-perl: 0.3.5

libpve-access-control: 8.2.2

libpve-apiclient-perl: 3.3.2

libpve-cluster-api-perl: 8.1.2

libpve-cluster-perl: 8.1.2

libpve-common-perl: 8.3.2

libpve-guest-common-perl: 5.2.2

libpve-http-server-perl: 5.2.2

libpve-network-perl: 0.11.2

libpve-rs-perl: 0.9.4

libpve-storage-perl: 8.3.6

libspice-server1: 0.15.1-1

lvm2: 2.03.16-2

lxc-pve: 6.0.0-1

lxcfs: 6.0.0-pve2

novnc-pve: 1.6.0-2

proxmox-backup-client: 3.4.3-1

proxmox-backup-file-restore: 3.4.3-1

proxmox-backup-restore-image: 0.7.0

proxmox-firewall: 0.7.1

proxmox-kernel-helper: 8.1.1

proxmox-mail-forward: 0.3.3

proxmox-mini-journalreader: 1.5

proxmox-offline-mirror-helper: 0.6.7

proxmox-widget-toolkit: 4.3.12

pve-cluster: 8.1.2

pve-container: 5.2.7

pve-docs: 8.4.0

pve-edk2-firmware: 4.2025.02-4~bpo12+1

pve-esxi-import-tools: 0.7.4

pve-firewall: 5.1.2

pve-firmware: 3.16-3

pve-ha-manager: 4.0.7

pve-i18n: 3.4.5

pve-qemu-kvm: 9.2.0-7

pve-xtermjs: 5.5.0-2

qemu-server: 8.4.1

smartmontools: 7.3-pve1

spiceterm: 3.3.0

swtpm: 0.8.0+pve1

vncterm: 1.8.0

zfsutils-linux: 2.2.8-pve1

 

[Uuugh]

Unfortunately, it appears that resolving this issue is not on Proxmox's agenda (note the length of time for this thread).
Like you, one day it worked, and then it didn't. I suspect this is a problem for many others using PCI passthrough for their GPUs; they just haven't realized yet that it is the cause of their system crashing.
My next step is to reformat and reinstall. That "magic seems to occasionally work.

 

[v95klima]

Successfull Workaround by rolling back kernel to 6.8.12-11-pve:

proxmox-boot-tool kernel list

promox-boot-tool kernel pin 6.8.12-11-pve

 

[kimus]

after almost... 1 year.. this guide fix (for me) the famous problem

error writing '1' to '/sys/bus/pci/devices/0000:c6:00.0/reset': Inappropriate ioctl for device

its a windows side, but for me it works, I don't have to restart the host anymore, win11 vm stop and restart like a charm
I hope it's useful for someone

 

[Uuugh]

Successfull Workaround by rolling back kernel to 6.8.12-11-pve:

proxmox-boot-tool kernel list

promox-boot-tool kernel pin 6.8.12-11-pve

apt update && apt install pve-kernel-6.8.12-11-pve
promox-boot-tool kernel pin 6.8.12-11-pve

and then you'll need to

update-initramfs -u -k all

Thanks again @v95klima !

 

[jtheisen]

I now also have this problem with a second server, running a 7900 XTX
Sadly i can't apply @Uuugh 's workaround since the cluster is already at PVE 9 and the 6.8 Kernel is not available for trixie ...

In my case the VM is running NixOS
Maybe this has something to do with the Guest OS not properly shutting down the GPU on shutdown?

Interestingly enough the VM also doesn't tolerate being snapshotted i.e. for a backup. It just freezes and i have to reboot the node.
Changing the backup mode to "Stop" also didn't help, as i then just couldn't restart the VM due to this error:

kvm: ../hw/pci/pci.c:1815: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed

---

Server sepcs:
EPYC 7443P
Supermicro H12SSL-i
AMD 7900 XTX

pveversion -v

proxmox-ve: 9.1.0 (running kernel: 6.17.2-1-pve)
pve-manager: 9.1.1 (running version: 9.1.1/42db4a6cf33dac83)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
proxmox-kernel-6.17: 6.17.2-1
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
proxmox-kernel-6.14: 6.14.11-4
amd64-microcode: 3.20250311.1
ceph: 19.2.3-pve2
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.4
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.0.15
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.3
libpve-rs-perl: 0.11.3
libpve-storage-perl: 9.0.18
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-3
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.20-1
proxmox-backup-file-restore: 4.0.20-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.2
pve-cluster: 9.0.7
pve-container: 6.0.18
pve-docs: 9.1.1
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.0.8
pve-i18n: 3.6.2
pve-qemu-kvm: 10.1.2-4
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.0
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1

 

[uzumo]

It's not that it can't be launched in environments migrated from PVE 8, so it's not that it can't be used, but it can't be obtained.

I suppose you could make it usable even now, but I wouldn't recommend sticking with an old kernel.

* It's impossible if you upgraded the zpool without reason.
* Make requires multiple packages.

mkdir /root/tmp/git/ -p
cd /root/tmp/git
rm -rf .git
git init
git clone git://git.proxmox.com/git/pve-kernel.git -b bookworm-6.8
make
dpkg -i *.deb
proxmox-boot-tool kernel pin 6.8.12-17-pve