[SOLVED] I need help setting up double gpu passthrough

The X570 (and X570S) chipset have proper IOMMU groups (so no override needed). Of course most PCIe and M.2 slots will still go through (and be limited by) the chipset.

EDIT: Turns out that the TUF GAMING X570-PLUS (WI-FI) does not have two x16 PCIe slots with x8 lanes (split from the x16 of the CPU) like many other motherboards with X570 (and X470/X370), unfortunately.
 
Last edited:
  • Like
Reactions: uzumo and brightrgb
The ASM1166 has some quirks in its firmware, and configuring the ROMBAR on the PVE9 is quite inconvenient.

Additionally, I believe the stability of NGFF M.2 SATA is low.


 
The ASM1166 has some quirks in its firmware, and configuring the ROMBAR on the PVE9 is quite inconvenient.

Additionally, I believe the stability of NGFF M.2 SATA is low.
Are other chips better than ASM1166?

Yes, but unfortunately, many devices of the big chipset group are accessible-from/have-access-to the Proxmox host (like the network, drive and USB controllers and more). Your VM can, in principle, read all of the host memory (and therefore all of the other VMs) and steal passwords and other data without you knowing it.
In other forums I see, that on newer motherboards there is an option "ACS enable" in BIOS.
Is enabling ACS in BIOS as unsecure as patching the kernel with the acs patch?
 
ACS needs to be enabled to give you IOMMU groups. You don't need to patch the Proxmox kernel.
This is to do to get IOMMU groups.
And in most cases you get a big IOMMU-chipset group with B550 boards.

Then you can brake this isolation an become virualized seperate IOMMU groups for the chipset group by patching the kernel with:
"quiet iommu=pt pcie_acs_override=downstream,multifunction"

Is this correct?
 
And in most cases you get a big IOMMU-chipset group with B550 boards.
You always get this because devices connected to/via B550 are not properly isolated.
Then you can brake this isolation an become virualized seperate IOMMU groups for the chipset group by patching the kernel with:
"quiet iommu=pt pcie_acs_override=downstream,multifunction"

Is this correct?
You are not patching the kernel. You are enabling the "break the groups" that is already in the Proxmox kernel.
This is unsafe because it makes it look like devices are isolated but they are not really.
 
  • Like
Reactions: brightrgb
You are not patching the kernel. You are enabling the "break the groups" that is already in the Proxmox kernel.
Thank you, that I don't know.

his is unsafe because it makes it look like devices are isolated but they are not really.
Thank you, that is clear to me. Even though I find it very difficult to accurately assess the risk for my specific use case.
In my opinion it is very good, that in this forum the mebers tell us, that it is risky.
In other forums the ACS patch is used lik a tutorial :(
 
Even though I find it very difficult to accurately assess the risk for my specific use case.
In my opinion it is very good, that in this forum the mebers tell us, that it is risky.
In other forums the ACS patch is used lik a tutorial :(
In principle software inside the VM can use the PCI(e) devices (passed through to the VM) to potentially read the all memory of the Proxmox host (via the devices still connected to the host).
Whether this actually works to read the host memory from inside the VM via DMA might depend on a lot of things.

If you don't run any untrusted software and no untrusted users/access inside the VM then it might not be a big deal. But running software from the internet and/or having internet access in the VM can be risky.

You need to do your own threat analysis and decide on what risks you want to take. Lots of people just don't bother and use the pcie_acs_override because they think the IOMMU groups are in the way, instead of realizing that they are there to protect them. But even with proper IOMMU group isolation, actual hardware passed through to the VM can take down the host when it crashes.
 
  • Like
Reactions: brightrgb
In principle software inside the VM can use the PCI(e) devices (passed through to the VM) to potentially read the all memory of the Proxmox host (via the devices still connected to the host).
Whether this actually works to read the host memory from inside the VM via DMA might depend on a lot of things.
One last question.
This only applies if the ACS patch is active, right?

If you passthrough a GPU to a VM using the standard IOMMU groups (without patching), then there's no risk. Right?
 
This only applies if the ACS patch is active, right?
Yes.
If you passthrough a GPU to a VM using the standard IOMMU groups (without patching), then there's no risk. Right?
I cannot guarantee that there is no risk, as passing real hardware to a VM can interfere with the host. But without the pcie_acs_override, IOMMU should protect you from devices secretly communicating (and reading memory).
 
  • Like
Reactions: brightrgb
I cannot guarantee that there is no risk, as passing real hardware to a VM can interfere with the host.
Sure, every system could have security issues.

But thanks to you, I now understand where the danger lies with ACS Override and when it might be acceptable.
Of course, only the user can make the final decision.