Proxmox 8 migration issue with pci passthrough (JMB585 sata adapter)

[migueljal]

Hello Proxmox community!
I wanted to bring your attention to a bug that I recently encountered when upgrading from Proxmox 7 to version 8.0. This bug specifically affects the functionality of a JMB585 device passthrough, causing it to stop working without any clear explanation.
Before the upgrade, I had successfully configured and utilized a JMB585 device passthrough on Proxmox 7, which was working flawlessly. However, upon completing the upgrade to Proxmox 8.0, the passthrough functionality ceased to function as expected.
After some investigation and troubleshooting, I discovered a temporary workaround. By reverting to Proxmox 8.0 but utilizing the kernel version 5.15, I was able to restore normal operation of the JMB585 device passthrough. This suggests that the issue lies within the compatibility between Proxmox 8.0 and the default kernel version it utilizes.
Although I am relieved to have found a solution, I believe it is important to bring this bug to the attention of the community and the Proxmox development team. As it stands, users upgrading to Proxmox 8.0 and relying on JMB585 device passthrough may face unexpected issues, and it would be beneficial to have a permanent resolution in place.
I kindly request the Proxmox development team to investigate this issue further and provide an official explanation and fix for this bug. This will help ensure a smooth transition and seamless functionality for users who heavily rely on JMB585 device passthrough.
If any fellow community members have encountered similar issues or have additional insights, please feel free to share your experiences in this thread. Collaboration and knowledge-sharing are vital in resolving such bugs effectively.
Thank you for your attention, and let's work together to address and resolve this issue promptly!
Best regards, Miguel

 

[leesteken]

I fear that passthrough cannot be guaranteed under any circumstances. Thank you for bringing this to the attention of the community. Can I please ask you to add JMB585 to the title of this thread to make it easier to find for people using that device? You might also want to file a bugreport at bugzilla.proxmox.com.

Before the upgrade, I had successfully configured and utilized a JMB585 device passthrough on Proxmox 7, which was working flawlessly. However, upon completing the upgrade to Proxmox 8.0, the passthrough functionality ceased to function as expected.

This appears to be cause by the newer kernel (as you discovered) and I don't know if Proxmox can fix it or the Linux kernel needs to investigate it. Can you describe how it "ceased to function"? Maybe it just needs a work-around or some additional configuration. Or maybe it's a IOMMU group issue and not specific to this device? Can you show the IOMMU groups with kernel 5.15 and with 6.2? Or maybe it's a reset issue and early binding it to vfio-pci resolved it? Can you explain what steps you took to passthrough that particular device?

 

[migueljal]

Apologies for the delay in responding. It simply stopped working after updating Proxmox. It was previously functioning using the official guides (https://pve.proxmox.com/wiki/PCI_Passthrough). Despite restarting the virtual machine several times to which the JMB585 was connected, it doesn't start. I'm attaching the IOMMU groups in kernel 6.2 and 5.15 with the dmesg messages:

Kernel 6.2
/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:16.0
/sys/kernel/iommu_groups/15/devices/0000:02:01.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.3
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.5
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
/sys/kernel/iommu_groups/18/devices/0000:39:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:17.0
/sys/kernel/iommu_groups/16/devices/0000:02:02.0
/sys/kernel/iommu_groups/6/devices/0000:00:15.2
/sys/kernel/iommu_groups/6/devices/0000:00:15.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.5
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.6
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/20/devices/0000:3b:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
/sys/kernel/iommu_groups/19/devices/0000:3a:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0

dmesg | grep 0000:3a:00.0
[ 0.844143] pci 0000:3a:00.0: [197b:0585] type 00 class 0x010601
[ 0.844164] pci 0000:3a:00.0: reg 0x10: [io 0x3200-0x327f]
[ 0.844175] pci 0000:3a:00.0: reg 0x14: [io 0x3180-0x31ff]
[ 0.844187] pci 0000:3a:00.0: reg 0x18: [io 0x3100-0x317f]
[ 0.844198] pci 0000:3a:00.0: reg 0x1c: [io 0x3080-0x30ff]
[ 0.844210] pci 0000:3a:00.0: reg 0x20: [io 0x3000-0x307f]
[ 0.844221] pci 0000:3a:00.0: reg 0x24: [mem 0x56210000-0x56211fff]
[ 0.844233] pci 0000:3a:00.0: reg 0x30: [mem 0x56200000-0x5620ffff pref]
[ 0.844326] pci 0000:3a:00.0: PME# supported from D3hot
[ 1.006429] pci 0000:3a:00.0: Adding to iommu group 19
[ 1.588185] ahci 0000:3a:00.0: SSS flag set, parallel bus scan disabled
[ 1.588230] ahci 0000:3a:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
[ 1.588233] ahci 0000:3a:00.0: flags: 64bit ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh
[ 106.261066] vfio-pci 0000:3a:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 106.322365] vfio-pci 0000:3a:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 106.322447] vfio-pci 0000:3a:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 107.048699] vfio-pci 0000:3a:00.0: timed out waiting for pending transaction; performing function level reset anyway
[ 108.296678] vfio-pci 0000:3a:00.0: not ready 1023ms after FLR; waiting
[ 109.352556] vfio-pci 0000:3a:00.0: not ready 2047ms after FLR; waiting
[ 111.624486] vfio-pci 0000:3a:00.0: not ready 4095ms after FLR; waiting
[ 115.980424] vfio-pci 0000:3a:00.0: not ready 8191ms after FLR; waiting
[ 124.424303] vfio-pci 0000:3a:00.0: not ready 16383ms after FLR; waiting



Kernel 5.15

/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:16.0
/sys/kernel/iommu_groups/15/devices/0000:02:01.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.3
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.5
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/18/devices/0000:39:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:17.0
/sys/kernel/iommu_groups/16/devices/0000:02:02.0
/sys/kernel/iommu_groups/6/devices/0000:00:15.2
/sys/kernel/iommu_groups/6/devices/0000:00:15.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.5
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.6
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/20/devices/0000:3b:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:3a:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0

dmesg | grep 0000:3a:00.0
[ 0.654090] pci 0000:3a:00.0: [197b:0585] type 00 class 0x010601
[ 0.654112] pci 0000:3a:00.0: reg 0x10: [io 0x3200-0x327f]
[ 0.654124] pci 0000:3a:00.0: reg 0x14: [io 0x3180-0x31ff]
[ 0.654135] pci 0000:3a:00.0: reg 0x18: [io 0x3100-0x317f]
[ 0.654146] pci 0000:3a:00.0: reg 0x1c: [io 0x3080-0x30ff]
[ 0.654157] pci 0000:3a:00.0: reg 0x20: [io 0x3000-0x307f]
[ 0.654168] pci 0000:3a:00.0: reg 0x24: [mem 0x56210000-0x56211fff]
[ 0.654179] pci 0000:3a:00.0: reg 0x30: [mem 0x56200000-0x5620ffff pref]
[ 0.654267] pci 0000:3a:00.0: PME# supported from D3hot
[ 0.759810] pci 0000:3a:00.0: Adding to iommu group 19
[ 1.296092] ahci 0000:3a:00.0: SSS flag set, parallel bus scan disabled
[ 1.296137] ahci 0000:3a:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
[ 1.296141] ahci 0000:3a:00.0: flags: 64bit ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh
[ 51.124082] vfio-pci 0000:3a:00.0: vfio_ecap_init: hiding ecap 0x19@0x300
[ 51.124086] vfio-pci 0000:3a:00.0: vfio_ecap_init: hiding ecap 0x1e@0x900

 

[leesteken]

Apologies for the delay in responding. It simply stopped working after updating Proxmox. It was previously functioning using the official guides (https://pve.proxmox.com/wiki/PCI_Passthrough). Despite restarting the virtual machine several times to which the JMB585 was connected, it doesn't start. I'm attaching the IOMMU groups in kernel 6.2 and 5.15 with the dmesg messages:

Looks like a regression in the Linux kernel. If Proxmox did not accidentally break the reset mechanism for that device then you'll probably need to report this upstream (if you can reproduce it without Proxmox and just using KVM).

 

[etnicor]

For anyone seeing this thread. Working fine for me on kernel 6.2.16-14

pcie passthrough to i440x vm.

 

[migueljal]

For anyone seeing this thread. Working fine for me on kernel 6.2.16-14

pcie passthrough to i440x vm.

It also doesn't work for me.

 

[etnicor]

It also doesn't work for me.

Weired
I assume you have blacklisted AHCI driver on host or prevented host to initiate the jmb585 device?

From within ubuntu 22.04 vm:

lspci |grep -i micron
00:1b.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller

This is the card I use.
https://www.aliexpress.com/item/1005004358183286.html