Proxmox Got freezed PC when GPU passthrough at PCI-e 16x (PCIEX4)

[gerardgarcia]

Hi there!

I have a Proxmox Server with an AMD Ryzen 7 pro, and a B450 Aorus M motherboard, a Sapphire Pulse RX580 8GB and a Nvidia GT710 1GB.
The motherboard got GPU passthrough successfully on the main PCI-e 16X speed (no matters which GPU I plug there).

But the issue comes when I'm trying to plug a GPU in the second port PCI-e that runs at 4x (PCIEX4).
No matters if I only plug one GPU in that port and nothing connected on the main port.
So, when I try to start a VM, using the GPU plugged on the PCIEX4 port, the computer got freezed instantly.

So the question is, will be a solution to got working GPU passthough on that PCIEX4 port using proxmox?
Or I might to resign to buy another motherboard that have two PCI-e 16X?

This is what Gigabyte manufacturer says about the specs of the mobo:
Expansion Ports:

1. 1 x PCI Express x16 slot, running at x16 (PCIEX16)
   * Actual support may vary by CPU.
   * For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.
   (The PCIEX16 slot conforms to PCI Express 3.0 standard.)
2. 1 x PCI Express x16 slot, running at x4 (PCIEX4)
3. 1 x PCI Express x1 slot
   (The PCIEX4 and PCIEX1 slots conform to PCI Express 2.0 standard.)

Thanks in advance

 

[gerardgarcia]

I tried to add pcie_acs_override=downstream,multifunction on etc/default/grub in order to split the IOMMU groups, but, when I do this, the GT710 dissapear from the list.

 

[leesteken]

So the question is, will be a solution to got working GPU passthough on that PCIEX4 port using proxmox?
Or I might to resign to buy another motherboard that have two PCI-e 16X?

Only the X570 and X570S have proper IOMMU isolation on all PCIe slots and (most) onboard devices. All other AM4 motherboards can only do passthough on the one x16 slot and the one M.2 slot (4 lanes) that use PCIe lanes directly from the CPU (and onboard devices that come from the CPU as well) . All other slots are in one big "chipset IOMMU group". The X470 and X370 can additionally split that x16 slot into two times x8 (which look like x16 for compatibility) that work with passthrough. Sometimes motherboard BIOS versions break passthrough as well.

 

[gerardgarcia]

Ok, I

Only the X570 and X570S have proper IOMMU isolation on all PCIe slots and (most) onboard devices. All other AM4 motherboards can only do passthough on the one x16 slot and the one M.2 slot (4 lanes) that use PCIe lanes directly from the CPU (and onboard devices that come from the CPU as well) . All other slots are in one big "chipset IOMMU group". The X470 and X370 can additionally split that x16 slot into two times x8 (which look like x16 for compatibility) that work with passthrough. Sometimes motherboard BIOS versions break passthrough as well

Ok I gonna try that.

As new trying, a few days I bought and PCI Express 2x8 Bifurcator, in order to divide the 16x lane in two 8x lanes and put two graphic cards on the Bus that supposedly work with IOMMU.

I set it up in BIOS, I have video on the RX580, but proxmox system can’t see the GT710.

Then I reboot the server, and plug the display and boot with the GT710.

But in certain way system it’s not freezed at all, but can’t send video to the RX580.

Then I access the proxmox server portal in my mobile, and I notice that in the list of IOMMU Peripherals it’s Empty! I mean it doesn’t detect any IOMMU Periferals.

So, I give up, trying with bifurcator and I gonna try with the x570 or x570s mobo and see what’s happen

 

[gerardgarcia]

I've solved it, Fortunatelly its not a hardware issue.
I use ACS multifunction on GRUB like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=multifunction clocksource=tsc tsc=reliable

Then update-grub, make sure that IOMMU Groups of both GPUs are separated and Yes.
Then Add the ID of both GPUS to VFIO.
Also I blacklisted all Video Drivers (Nvidia, AMD, Radeon, Intel).
Then Reboot and assign a GPU to each VM and It Works.

But, I found an Issue, I noticed that If the GPU has not plugged a monitor, the VM wont start.
So I turn on a VM with a monitor plugged to its GPU, once turned on, then I plugged the monitor to the another GPU attached to the another VM and thats it!!!

Both Virtual Machines turned on an running at the same time.
I've read that in order to doing this each time that I need to run the VMs, I need to buy an HDMI Dummy.

I dont know if theres a way or config that I can use on config file of the VMs that makes unneccesary to buy the HDMI Dummy.



Also as you noticed, I'm using clocksource=tsc and on macOS VM using "Haswell-noTSX", because before that doing this the macOS system doesnt run, reboots after shows the Apple Logo in an endless loop.

I discovered that on another forum that modern processors doesnt manage TSC so well, so that flags are needed