GPU passthrough (Pascal) broken on PVE9 kernels, fixed by pinning 6.8.12-43-pve — data point + request for insight
My hardware:
- Motherboard: Gigabyte GA-Z97X-UD3H (reported by system as Z97X-UD3H-CF), BIOS F9 (2015-09-18)
- CPU: Intel Xeon E3-1245 v3 @ 3.40GHz (Haswell)
- GPUs passed through (full PCI passthrough, VFIO, not vGPU):
- NVIDIA Quadro P2000 (GP106GL), PCI ID 10de:1c30 + audio 10de:10f1 → Ubuntu VM
- NVIDIA GeForce GT 1030 (GP108), PCI ID 10de:1d01 + audio 10de:0fb8 → Windows Server 2019 VM
- Kernel cmdline: intel_iommu=on iommu=pt pcie_aspm=off pcie_acs_override=downstream,multifunction
Summary:
Both GPUs (both Pascal architecture) worked flawlessly under PVE8 (kernel 6.8.12-22-pve) for years, full passthrough to two separate VMs. After apt dist-upgrade from PVE8 to PVE9 (ending up on kernel 7.0.14-12-pve), passthrough broke for
both GPUs simultaneously, with different guest-side symptoms:
- Windows VM (GT 1030): Device Manager shows Code 43 on the NVIDIA adapter.
- Linux VM (Quadro P2000): nvidia_drm/nvidia_modeset/nvidia_uvm/nvidia modules all load successfully, device is enumerated correctly via lspci inside the guest, but nvidia-smi reports "No devices were found". dmesg shows:
Code:
NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x31:0xffff:2780)
NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
Identical failure across NVIDIA driver versions 570, 580, and 580-open — ruling out a simple driver regression.
What was ruled out:
- IOMMU / interrupt remapping: confirmed enabled (DMAR-IR: Enabled IRQ remapping in xapic mode).
- IOMMU groups: Quadro P2000 + its audio function sit cleanly in their own IOMMU group (group 2), separate from other host devices. No ACS override needed for this reason.
- VFIO binding and reset: clean in host dmesg for both GPUs, e.g.:
Code:
vfio-pci 0000:01:00.0: resetting
vfio-pci 0000:01:00.0: reset done
vfio-pci 0000:05:00.0: resetting
vfio-pci 0000:05:00.0: reset done
No unable to reset, not ready, or stuck in D3 messages at any point — ruling out the classic Pascal consumer-card FLR/reset issue as the cause here.
- cpu: host,hidden=1 (hiding hypervisor from guest): applied to both VMs. No change in either symptom.
- Custom VBIOS via romfile: dumped a reference GP106 VBIOS from TechPowerUp and applied via hostpci0: ...,romfile=GP106-P2000.rom (previously rombar=0). No change — identical 0x31:0xffff:2780 error, same offset even.
What actually fixed it:
Pinning the host to the last working PVE8-era kernel (proxmox-boot-tool kernel pin 6.8.12-43-pve, the closest available successor to the 6.8.12-22-pve that was running under PVE8) resolved
both VMs immediately, with no other config changes reverted (hidden=1 and romfile were left in place, unnecessary but harmless):
Code:
$ nvidia-smi
Driver Version: 580.173.02 CUDA Version: 13.0
GPU 0 Quadro P2000 ... (working normally)
Windows VM: Code 43 also gone, GT 1030 working normally in Device Manager.
Kernel is the only variable that changed between "broken" and "working" across all these tests.
Why I don't think that this is a single-card fault:
Both affected GPUs are different chips (GP106 vs GP108), different board partners even, yet failed identically and were fixed identically by the same kernel change. This makes an individual card defect very unlikely, and points at something architecture-level (Pascal) and/or platform-level (this specific Haswell/Z97 board) interacting badly with kernels ≥ ~6.14/7.0.
A relevant negative control:
A completely different machine (HP ProLiant DL360 G5, Xeon 5100/5300 era, ~2007-2008) passes through a GeForce GT 710 (GK208B,
Kepler architecture, one generation older than Pascal) without any issue on the same PVE9 / kernel 7.0 series. This rules out "PVE9 in general" or "old board in general" as sufficient explanations on their own — it seems more specific to Pascal-generation NVIDIA silicon combined with this consumer-grade Haswell platform (or possibly Pascal alone; I don't have a way to test a Pascal card in server-grade hardware or a post-Pascal card in this board to fully disentangle the two).
My (unconfirmed) working theory:
Newer kernels have become stricter about PCIe power-state transitions (ASPM, D3/D3cold, runtime PM) and rely on the board's ACPI tables (_PR0/_PR3 etc.) correctly describing power resources for each PCIe slot. A 2014 consumer board never validated for this kind of passthrough power-cycling could plausibly have incomplete/buggy ACPI power descriptors for its PCIe slots — invisible under bare-metal Windows/Linux (which never power-cycles the GPU this way), but exposed once a hypervisor + newer kernel enforces the full PCIe power handshake more strictly. This would explain why vfio-pci sees a clean reset (that layer doesn't care about deep power states) while NVIDIA's RmInitAdapter, which does a much deeper handshake with the silicon, fails.
This is speculation on my part — I have not verified it against kernel source or ACPI dumps. I'd be very interested if anyone has tooling or pointers to confirm/refute this (e.g. acpidump comparison, specific ASPM/PM kernel changelogs between 6.8 and 6.14/7.0 relevant to PCIe slot power resources).
Questions for the community / Proxmox team
- Is this a tracked/known issue specifically for Pascal-generation NVIDIA GPUs on kernels ≥ 6.14?
- Has anyone reproduced this on server-grade boards with Pascal cards (to help separate "Pascal architecture issue" from "consumer Haswell board issue")?
- Any known BIOS-level settings (PCIe link speed forcing, ASPM, ErP) that have resolved similar cases on consumer boards of this generation?
