[SOLVED] I need help setting up double gpu passthrough

[uzumo]

The key point is that no motherboards exist that split a 16-lane slot into two 8-lane slots, except for high-end chipsets. (It may exist, but it is rare.)

The Ryzen CPU lanes are 24 lanes for the 5700X3D. Four of these lanes are reserved for the connection between the chipset and the CPU and are not available to the user.

*On the motherboard, the CPU lanes are designed to be used regardless of the chipset.

The remaining 20 lanes can be used in configurations such as x8x4x4x4, x8x8x4, or x16x4. (Depending on the motherboard or BIOS design)

If there are no iommu issues and you are not dissatisfied with x4 connections, the x16x4 configuration is actually perfectly fine.

*Multiple people are probably playing using an eGPU like the x4 connection (Oculink)

*ECC support is complicated, so I don't know.

 

[leesteken]

Thank you, is B450 maybe an option or are the equal to B550?

No:

If you want to passthrough two PCIe slots (both working in x8 mode) you need a motherboard with a X570(S), X470 or X370 chipset.

 

[brightrgb]

No:

It's really crazy.
The MSI B550-A Pro seems to be the only board that has two PCIe slots connected to the CPU and not End Of Life.
Unfortunately, for some reason, the MSI doesn't support ECC.

I'd even be happy with a second PCIe slot, even just x4, but I can't find a B550 board with that.

 

[leesteken]

The MSI B550-A Pro seems to be the only board that has two PCIe slots connected to the CPU and not End Of Life.
Unfortunately, for some reason, the MSI doesn't support ECC.

I'd even be happy with a second PCIe slot, even just x4, but I can't find a B550 board with that.

Only the first PCIe x16 slot and the first M.2 slot (which is like PCIe x4) are usually connected to the CPU and can be used for PCI(e) passthrough. What makes you think the second slot will put a GPU (or other PCIe device) in a separate IOMMU group (without using the pcie_acs_override)? Only the X?70 chipsets support two PCIe x61 slots (both at x8) as far as I know.

 

[brightrgb]

Only the first PCIe x16 slot and the first M.2 slot (which is like PCIe x4) are usually connected to the CPU and can be used for PCI(e) passthrough.

In the manual of the ASRock B550 PG Riptide is written, that 3 PCIe are connected to the CPU.
Is this electrical possible with B550?
If yes, then I maybe give the board a try .....

 

[leesteken]

In the manual of the ASRock B550 PG Riptide is written, that 3 PCIe are connected to the CPU.
Is this electrical possible with B550?

It is not possible with the 24 PCIe lanes from the AM4 socket. Where does it say that it is connected to the CPU? The website only shows which slots with which speed and lanes works when which Ryzen family is installed. It does not say how they are connected?
Why keep hoping that you can find one motherboard that is special and wired completely different than all other AM4 mmotherboards? They all use the same chipsets with known limitations.

Maybe just use the ()built-in) pcie_acs_override if you insist on using a chipset that does not support (in a secure way) two PCIe slots in separate IOMMU groups? Although breaking the groups with the override does not guarantee that it works, but as you can see from old threads about AM4 and "the big chipset IOMMU group) it usually works.

EDIT: I didn't intend to sound so harsh. It's just that most AM4 chipsets don't support what you want and the chipsets that you can apparently buy simply don't support two passthrough devices securely. But you can, if you accept the risk, use the override on any chipset (with at least two PCIe slots that fit your devices). In my country, X570 motherboards (which are the best for passthrough) are still available.

 

[brightrgb]

Where does it say that it is connected to the CPU? The website only shows which slots with which speed and lanes works when which Ryzen family is installed. It does not say how they are connected?

In the manual:

AMD Ryzen series CPUs (Vermeer and Matisse)
• 3 x PCI Express x16 Slots (PCIE1: Gen4x16 mode; PCIE3:Gen3 x4 mode; PCIE4: Gen3 x1 mode)

In my country, X570 motherboards (which are the best for passthrough) are still available.

Cool, is it possible, that you send me an link of an X570 board offer.
Maybe I can import it in my country.

 

[leesteken]

In the manual:

It only states what speeds and lanes are available for each slot when using Vermeer or Matrisse. It does not specify that they are connected to the CPU (nor that they are in a separate IOMMU group).

Cool, is it possible, that you send me an link of an X570 board offer.

https://tweakers.net/moederborden/v...WmVOSWlSsZFWtZAw0GUiVJeYoWUUrGRqYmBorxdbW1gIA
ASRock X570S PG Riptide, ASUS ROG Crosshair VIII Impact, ASRock Rack X570D4U, Gigabyte X570 UD, ASRock Rack X570D4U-2L2T/BCM, ASRock Rack X570D4I-2T, ASUS ROG Crosshair VIII Extreme. From these, I would go with a ASRock Rack or maybe a Gigabyte as they often support choosing the boot GPU/PCIe-slot.

EDIT: As people are upgrading to AM5, aren't there any decent second-hard AM4 motherboards locally?

 

[brightrgb]

From these, I would go with a ASRock Rack or maybe a Gigabyte as they support chosing the boot GPU/PCIe-slot.

Thank you, I will see if an import will be possible.

EDIT: As people are upgrading to AM5, aren't there any decent second-hard AM4 motherboards locally?

Of course, but I've had bad experiences buying used items twice already!

 

[brightrgb]

Maybe just use the ()built-in) pcie_acs_override if you insist on using a chipset that does not support (in a secure way) two PCIe slots in separate IOMMU groups? Although breaking the groups with the override does not guarantee that it works, but as you can see from old threads about AM4 and "the big chipset IOMMU group) it usually works.

I've spent several hours now looking into the ACS patch.
Initially, I had serious security concerns, but if I understand correctly, the risk depends on many factors.

If I use the patch, than the memory management group is ignored and every PCI device is in its own group.
If I now passthroug a PCIe device that before was for example in the big chipset Group, then the guest VM can now access memory addresses of the real IOMMU group (all the memory that is in the big chipset group).
If no other device in the big chipset group is used in another VM or from the host, than the system is as secure as without the patch?

 

[leesteken]

If I now passthroug a PCIe device that before was for example in the big chipset Group, then the guest VM can now access memory addresses of the real IOMMU group (all the memory that is in the big chipset group).
If no other device in the big chipset group is used in another VM or from the host, than the system is as secure as without the patch?

Yes, but unfortunately, many devices of the big chipset group are accessible-from/have-access-to the Proxmox host (like the network, drive and USB controllers and more). Your VM can, in principle, read all of the host memory (and therefore all of the other VMs) and steal passwords and other data without you knowing it.

EDIT: If you can pass all devices from the big chipset group to one VM then you would be fine (or just pass some devices as the Proxmox host loses all devices from the IOMMU group when you passthrough some of them to a VM).