[SOLVED] I need help setting up double gpu passthrough

The key point is that no motherboards exist that split a 16-lane slot into two 8-lane slots, except for high-end chipsets. (It may exist, but it is rare.)

The Ryzen CPU lanes are 24 lanes for the 5700X3D. Four of these lanes are reserved for the connection between the chipset and the CPU and are not available to the user.

*On the motherboard, the CPU lanes are designed to be used regardless of the chipset.

The remaining 20 lanes can be used in configurations such as x8x4x4x4, x8x8x4, or x16x4. (Depending on the motherboard or BIOS design)

If there are no iommu issues and you are not dissatisfied with x4 connections, the x16x4 configuration is actually perfectly fine.

*Multiple people are probably playing using an eGPU like the x4 connection (Oculink)

*ECC support is complicated, so I don't know.
 
Last edited:
leesteken said:
No:
Click to expand...
It's really crazy.
The MSI B550-A Pro seems to be the only board that has two PCIe slots connected to the CPU and not End Of Life.
Unfortunately, for some reason, the MSI doesn't support ECC. :(

I'd even be happy with a second PCIe slot, even just x4, but I can't find a B550 board with that. :(
 
brightrgb said:
The MSI B550-A Pro seems to be the only board that has two PCIe slots connected to the CPU and not End Of Life.
Unfortunately, for some reason, the MSI doesn't support ECC. :(

I'd even be happy with a second PCIe slot, even just x4, but I can't find a B550 board with that. :(
Click to expand...
Only the first PCIe x16 slot and the first M.2 slot (which is like PCIe x4) are usually connected to the CPU and can be used for PCI(e) passthrough. What makes you think the second slot will put a GPU (or other PCIe device) in a separate IOMMU group (without using the pcie_acs_override)? Only the X?70 chipsets support two PCIe x61 slots (both at x8) as far as I know.
 
Last edited:
  • Like
Reactions: brightrgb
brightrgb said:
In the manual of the ASRock B550 PG Riptide is written, that 3 PCIe are connected to the CPU.
Is this electrical possible with B550?
Click to expand...
It is not possible with the 24 PCIe lanes from the AM4 socket. Where does it say that it is connected to the CPU? The website only shows which slots with which speed and lanes works when which Ryzen family is installed. It does not say how they are connected?
Why keep hoping that you can find one motherboard that is special and wired completely different than all other AM4 mmotherboards? They all use the same chipsets with known limitations.

Maybe just use the ()built-in) pcie_acs_override if you insist on using a chipset that does not support (in a secure way) two PCIe slots in separate IOMMU groups? Although breaking the groups with the override does not guarantee that it works, but as you can see from old threads about AM4 and "the big chipset IOMMU group) it usually works.

EDIT: I didn't intend to sound so harsh. It's just that most AM4 chipsets don't support what you want and the chipsets that you can apparently buy simply don't support two passthrough devices securely. But you can, if you accept the risk, use the override on any chipset (with at least two PCIe slots that fit your devices). In my country, X570 motherboards (which are the best for passthrough) are still available.
 
Last edited:
  • Like
Reactions: brightrgb
leesteken said:
Where does it say that it is connected to the CPU? The website only shows which slots with which speed and lanes works when which Ryzen family is installed. It does not say how they are connected?
Click to expand...
In the manual:
AMD Ryzen series CPUs (Vermeer and Matisse)
• 3 x PCI Express x16 Slots (PCIE1: Gen4x16 mode; PCIE3:Gen3 x4 mode; PCIE4: Gen3 x1 mode)
Click to expand...
leesteken said:
In my country, X570 motherboards (which are the best for passthrough) are still available.
Click to expand...
Cool, is it possible, that you send me an link of an X570 board offer.
Maybe I can import it in my country. :)
 
brightrgb said:
In the manual:
Click to expand...
It only states what speeds and lanes are available for each slot when using Vermeer or Matrisse. It does not specify that they are connected to the CPU (nor that they are in a separate IOMMU group).
brightrgb said:
Cool, is it possible, that you send me an link of an X570 board offer.
Click to expand...
https://tweakers.net/moederborden/v...WmVOSWlSsZFWtZAw0GUiVJeYoWUUrGRqYmBorxdbW1gIA
ASRock X570S PG Riptide, ASUS ROG Crosshair VIII Impact, ASRock Rack X570D4U, Gigabyte X570 UD, ASRock Rack X570D4U-2L2T/BCM, ASRock Rack X570D4I-2T, ASUS ROG Crosshair VIII Extreme. From these, I would go with a ASRock Rack or maybe a Gigabyte as they often support choosing the boot GPU/PCIe-slot.

EDIT: As people are upgrading to AM5, aren't there any decent second-hard AM4 motherboards locally?
 
Last edited:
  • Like
Reactions: brightrgb
leesteken said:
Maybe just use the ()built-in) pcie_acs_override if you insist on using a chipset that does not support (in a secure way) two PCIe slots in separate IOMMU groups? Although breaking the groups with the override does not guarantee that it works, but as you can see from old threads about AM4 and "the big chipset IOMMU group) it usually works.
Click to expand...
I've spent several hours now looking into the ACS patch.
Initially, I had serious security concerns, but if I understand correctly, the risk depends on many factors.

If I use the patch, than the memory management group is ignored and every PCI device is in its own group.
If I now passthroug a PCIe device that before was for example in the big chipset Group, then the guest VM can now access memory addresses of the real IOMMU group (all the memory that is in the big chipset group).
If no other device in the big chipset group is used in another VM or from the host, than the system is as secure as without the patch?
 
brightrgb said:
If I now passthroug a PCIe device that before was for example in the big chipset Group, then the guest VM can now access memory addresses of the real IOMMU group (all the memory that is in the big chipset group).
If no other device in the big chipset group is used in another VM or from the host, than the system is as secure as without the patch?
Click to expand...
Yes, but unfortunately, many devices of the big chipset group are accessible-from/have-access-to the Proxmox host (like the network, drive and USB controllers and more). Your VM can, in principle, read all of the host memory (and therefore all of the other VMs) and steal passwords and other data without you knowing it.

EDIT: If you can pass all devices from the big chipset group to one VM then you would be fine (or just pass some devices as the Proxmox host loses all devices from the IOMMU group when you passthrough some of them to a VM).
 
Last edited:
  • Like
Reactions: brightrgb
leesteken said:
EDIT: As people are upgrading to AM5, aren't there any decent second-hard AM4 motherboards locally?
Click to expand...
I had the change to get a cheap x570 over a forum.
So i bought a ASUS X570 TUF gaming plus Wifi.
I hope, that with this board a dual GPU passthrough will work in combination with pasthrough of an PCIe HBA.
 
The manual states that IOMMU settings themselves exist.

https://dlcdnta.asus.com/pub/ASUS/m...ME_PRO_TUF_GAMING_X570_Series_BIOS_EM_WEB.pdf

There is no block diagram, but on the X570, x16 lanes are only provided from the CPU.

Since it states that x16 lanes can be split into two x8 lanes, both x8 lanes are connected to the CPU.

https://dlcdnta.asus.com/pub/ASUS/m..._TUF_GAMING_X570-PLUS_WI-FI_UM_WEB_060319.pdf

Because IOMMU can be enabled on the CPU-connected lanes, it should behave as you expected.
 
uzumo said:
Since it states that x16 lanes can be split into two x8 lanes, both x8 lanes are connected to the CPU.

https://dlcdnta.asus.com/pub/ASUS/m..._TUF_GAMING_X570-PLUS_WI-FI_UM_WEB_060319.pdf

Because IOMMU can be enabled on the CPU-connected lanes, it should behave as you expected.
Click to expand...
Ohh, I think this is not so ... :(
The second PCIe x16 is only an x4 from X570 chipset. And in the manual I don't find, that I can split the 16 lines to 8/8.
So I guess I now have one of the few X570 boards where it won't work.

I just have no luck buying used. :(
 
Upon closer inspection, it's definitely x4 from the chipset...

My mistake, sorry.

I really need a block diagram...
 
  • Like
Reactions: brightrgb
uzumo said:
Upon closer inspection, it's definitely x4 from the chipset...

My mistake, sorry.
Click to expand...
Thank you so much.
It was my mistake to buy it.

Now I don't want to buy used anymore and will buy a B550 and patch it with ACS.
Are all B550s equally compatible with the ACS patch, or are there differences in how well the patch works?
 
We have confirmed operation on the A520. It should also work on the B550.

If the IOMMU group on the chipset's PCIe lanes matches other devices, setting ACS should allow it to function without issues.
 
  • Like
Reactions: brightrgb
uzumo said:
We have confirmed operation on the A520. It should also work on the B550.
Click to expand...
Thank you.
Or I only use one GPU for passthrough.

And use a M.2 to SATA adapter in an M.2 slot connected to the CPU for passthrough this to an TrueNAS VM.
This passthough should even work on B550 withouth ACS-patch, because it is connected to the CPU.
Is this right?
 
You must log in or register to reply here.
Top Bottom