Urgent Help vfio-pci 0000:03:00.0: can't change power state NVIDIA 4060 - X670E

[thebuildingguy]

Hi My Hardware

ASUS PROART X670E
AMD RYZEN 7900X 12 CORE
64GB DDR5 4800MHZ RAM

Bios - Latest October, 2023

RESIZABLE BAR - OFF
PCI Version 4
PCI LANE BANDWITH X8 X8 for PCI 1 & PCI 2


Proxmox

Proxmox 7.4.-17 (i tried 8.1 same issue and rolled back to 7.4)
pve-manager/7.4-17/513c62be (running kernel: 5.15.131-2-pve)


Basically i have 2 X 4060,

GPU 1 on PCI 1 works fine,

GPU 2 on PCI 2 is unaccessble if i do vfio.

The gpu is not accessible at all forget about passthrough.

I have setup everything like blacklist and all

Grub - GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt textonly nomodeset video=efifbff video=simplefbff video=vesafbff"


vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+memwns=none
[ 84.589746] pci 0000:01:00.0: Removing from iommu group 13
[ 84.609813] pci 0000:01:00.1: Removing from iommu group 13
[ 84.609910] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+memwns=none
[ 84.609913] vfio-pci 0000:03:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[ 84.609973] pci 0000:03:00.0: Removing from iommu group 15
[ 84.610007] vfio-pci 0000:03:00.1: can't change power state from D3hot to D0 (config space inaccessible)
[ 84.611607] pci 0000:03:00.1: Removing from iommu group 15

Please help.

I have a X570E ProArt which works fine.


Update: 1080 X 2 GPU (PCI 3) works fine strangely, one 4060 (first slot) and 1080 (second slot) also works fine
But as soon as the 4060 is placed in second slot, the above error comes again

 

[Petalzu]

I also encountered this problem today, and I tried many methods, which worked:

https://forum.proxmox.com/threads/v...power-state-currently-in-d3.60846/post-317760

There is actually a vfio-pci module parameter disable_idle_d3 which can be set to ON

modinfo vfio-pci

edit the file in /etc/modprobe.d where you have configured vfio-pci and append
disable_idle_d3=1

reboot, then check the dmesg output, now you should see normal mention of d3

dmesg | grep -i d3

So, in my understanding, this is not a good solution as it means that the PCI device will not be able to enter low-power mode. But it can temporarily solve the problem (if you don't mind power consumption).

Add some details:
/etc/modprobe.d is a folder where I modified a file named disable-nouveau.conf

Although this method is useful for my NVIDIA device, I cannot guarantee that it will solve your problem. You can try it out.

 

[thebuildingguy]

Yea this works disable_idle_d3=1 and i am using this but at the cost of power :>

 

[KrisFromFuture]

Yea this works disable_idle_d3=1 and i am using this but at the cost of power :>

can you have info about difference in power usage with D3 states OFF ?

 

[thebuildingguy]

Idle : 30-40 watts (4060) compare to 10 watt on d3 state on