Hardware upgrade can no longer pass through pcie (Unkown Header type 7f)

[Jamie]

Old hardware:

• Asus rog x570 crosshair viii hero (wi-fi)
• 5950x
• 128gb - 3200
• 1080ti x2
• sas hba

New hardware:

• Asus ProArt B650-Creator AM5 (BIOS AGESA 1.0.0.7a NOT b yet - soon)
• 7950x
• 128gb - 4200
• 1080ti x2
• sas hba

Both motherboard configurations have 3 pcie slots (8x 8x 4x) when all three are fitted
Both are using all the same hardware except motherboard cpu and ram. Proxmox boot disk is the same. So configuration is same.
My old hardware works perfectly with no issues
New hardware is giving me some troubles

Goal:
Pass all three pcie devices to three vms
Passing through of the second slot works fine. It's just the first is the issue.

Current Problem:
When booting any vm with the pcie device in (slot 1) it causes the pcie device to no longer function and I receive the error below: (lspci -v | more)



As you can see the iommu groups are isolated



Troubleshooting:

1. Bios
   1. above 4g : ENABLED
   2. resizeable bar: DISABLED
   3. Slot 1 Bifabrication disabled (8x mode with pcie slot 2 also 8x mode)
   4. Primary GPU: IGPU

I've searched more in depth in the bios and I cant seem to find any more settings for pcie.


Does anyone have any ideas? Thanks for reading

 

[Quaan]

M.2_3 slot shares bandwidth with pcie_3 slot ? i guess. There is a nvme in M.2 slot? If it is not the problem, please forgive me. I am not the professor here.

 

[leesteken]

I've see such a problem with motherboard BIOS versions (on a ASRock X470) that broke PCIe passthrough. Earlier or later BIOS versions resolved it.

 

[Jamie]

I returned the board and went with an x670e version of the same board. Is working now. Maybe in the future a bios update will fix the B650 board.

 

[derfreak]

I have the exact same issue with the Asus ProArt B650-Creator after upgrading from the initial bios version 0213 to the latest version as of this post (1807).


I had three pcie devices which where passed through to two virtual machines - everything was nice.
After I upgraded the bios I can no longer pass through the GTX 1050 Ti in Slot #1.

Does anyone know which version except from the initial bios works ?

 

[derfreak]

I rolled back my Bios to 0213 and can confirm that passthrough works again for all PCIe slots !
Starting from 1807 I tested every Bios down to 1303 - all of these do not work.
I lost patience and went back to 0213 which worked.

It would be interesting to see which version introduced the issue but I don't want to flash my bios again.

 

[earthy]

I have same problem as you. I tested remaining bioses (0213 - 1101, 1807, 1904) and can confirm 2xGPU passthrough works only on bios 0213.
I asked Asus support for help and am waiting to hear back from them.

 

[derfreak]

I did the same !
The latest update was that they passed the information on to the developers, let's hope that they find something.

 

[derfreak]

@earthy
Sadly I never heard back from them, did you ?

In the meantime they released 1905 and 2007 but the patchnotes don't mention our issue.