Motherboard for GPU passthrough

[earthy]

Hello,
I am building new box with ProxMox but am stuck at motherboard.
The tricky part is which motherboard to buy for two GPUs (NVIDIA) to present into one VM for playing with LLMs.

Other aspects of configuration and use cases are somewhat straightforward (VMs, SMB). Setup so far, without motherboard:
AMD Ryzen 9 7900
64GB RAM (ECC if possible)
2x m.2 NVME for fast storage
2x HDD for slow storage
2x NVIDIA graphic cards

Did not find a list of unproblematic motherboards or what works well with ProxMox and gpu passthrough.

Help would be greatly appreciated.

 

[leesteken]

X570(S) has the best IOMMU groups (and can often passthrough at least 3 GPUs) and X470/X370 usually has two PCIe slots (and a M.2) for passthrough. Other AM4 chipsets can only passthrough one PCIe slot and one M.2 slot. I don't know the status of AM5 but I fear it's not better than one PCIe slot and one M.2...
ECC is supported by Ryzen but check the motherboard specifications if it explicitly supports it.

 

[earthy]

Thank you, but Ryzen 9 7900 has AM5 socket, so above chipsets do not apply here.

 

[leesteken]

Thank you, but Ryzen 9 7900 has AM5 socket, so above chipsets do not apply here.

I know, I just wanted to warn you that passthrough of more than 1 GPU might not be possible with AM5.

EDIT: Might be a good idea to check the Level1techs forum: https://forum.level1techs.com/t/am5-iommu-vfio-best-motherboard/189720

 

[Dunuin]

And keep the thickness of the GPUs in mind. You are limited to the 1 or 2 PCIe 16x slots and especially the bigger GPUs might be tripple-slot and won't perform well if there is no additional space between them for airflow. So ideally with 16x slots at position 1 + 4 or even 5.

 

[earthy]

Thank you for the link @leesteken!
From that thread, ASUS ProArt B650 Creator looks like capable motherboard with good IOMMU groupings. Certified for ECC RAM, 2 NICs (Realtek, but oh well) are nice additions.
Bonus is that it is not that expensive.

I see i am perhaps wanting too much with two GPUs to passthrough, but lets see how it goes.

If/when i get it and set it up, i will report back (if there is anything to report).

Thank you all for prompt replies.

 

[Dunuin]

There is also https://iommu.info/ if you want to know what IOMMU groups a mainboard is offering.

 

[leesteken]

From that thread, ASUS ProArt B650 Creator looks like capable motherboard with good IOMMU groupings.

Where can I find the IOMMU groups?

 

[earthy]

@Dunuin posted your reply. Here it is: https://iommu.info/mainboard/152

 

[leesteken]

@Dunuin posted your reply. Here it is: https://iommu.info/mainboard/152

I didn't realize you looked it up before they posted their very helpful link. I doubt that those are the actual groups (without using pcie_acs_override) but maybe I'm just cynical (and not well informed) and you found a great motherboard. I do hope you are right and would be happy to be proven wrong.

 

[MichaelBernasconi]

I have a Gigabyte X670 GAMING X AX and GPU passthrough is working well. Passing through the GPU in the main slot is possible even without pcie_acs_override. I posted a detailed description of how I got my setup working here.

 

[earthy]

Just a status update.
System is set up and running. 2xGPU passthrough is working normally (on ASUS ProArt B650 Creator).
IOMMU grups are good out of the box: PCIe_1 and PCIe_2 are in seperate groups and do not share groups with anything else.

BUT only after i downgraded bios to first version... all the newer versions break support for 2xGPU passthrough... Groups are good, but GPU in slot PCIe_1 is just not aresponding correctly. Same problem described by other users:
https://forum.proxmox.com/threads/h...cie-unkown-header-type-7f.131123/#post-625340

Waiting for Asus to respond to my question about this.

 

[skar]

X570(S) has the best IOMMU groups (and can often passthrough at least 3 GPUs) and X470/X370 usually has two PCIe slots (and a M.2) for passthrough. Other AM4 chipsets can only passthrough one PCIe slot and one M.2 slot. I don't know the status of AM5 but I fear it's not better than one PCIe slot and one M.2...
ECC is supported by Ryzen but check the motherboard specifications if it explicitly supports it.

hello may i ask about the x570m (ASROCK X570M Pro4) I would like to build a Ryzen 5700G pc and passthrough a Radeon rx6600 do you think it's a fitting choice ?

 

[_--James--_]

hello may i ask about the x570m (ASROCK X570M Pro4) I would like to build a Ryzen 5700G pc and passthrough a Radeon rx6600 do you think it's a fitting choice ?

Should be good, I have a Asrock B550M-Pro4 with a 5800X3d and RX6600X with IOMMU setup and it works fine. Just make sure your PCIE slot is set for x16 and not auto for the bifurcation options under CBS.