[SOLVED] I need help setting up double gpu passthrough

[Hiroi]

Hi! I recently have downloaded proxmox on my pc and I have to some problems with gpu passthough.
For context, I have a amd 5800X cpu paired with a rtx 2060s and a amd RX540.
What I want to do it to run the server headless and control it through the web browser and have the other 2 gpus passed to 2 independent vms. With ACS override, I managed to have everything in separate groups so everything was fine there.

Now everything works fine with one gpu passed through, but when I start the second vm with the second gpu both vms freeze and after a bit the whole proxmox server goes down, I then have to do a fsck and then I can get into my server again. I tested both gpus separately and the both worked like a charm, also this only happens when I have the gpus passed at the same time since I did run a couple vms at the same time when I only 1 gpu passed and it was fine. I am not sure what could cause this, in the configs I only touched the gpus, nothing more yet it crashes my whole server. I would really appreciate some help.

Thanks!

 

[dcsapak]

Now everything works fine with one gpu passed through, but when I start the second vm with the second gpu both vms freeze and after a bit the whole proxmox server goes down, I then have to do a fsck and then I can get into my server again. I tested both gpus separately and the both worked like a charm, also this only happens when I have the gpus passed at the same time since I did run a couple vms at the same time when I only 1 gpu passed and it was fine. I am not sure what could cause this, in the configs I only touched the gpus, nothing more yet it crashes my whole server. I would really appreciate some help.

this sounds very much like the gpus are in the same iommu group

With ACS override, I managed to have everything in separate groups so everything was fine there.

this only forces seperated iommu groups in the kernel, it does not change the limitations of the motherboard. some times it can work, but often it just makes things worse

check if your gpus are in the same groups (with the ACS patch disabled). if they are, my guess is that there is not much you can do besides changing the underlying hardware (mostly mobo+cpu)

 

[leesteken]

Please check your IOMMU groups without pcie_acs_override and tell us exactly which motherboard you are using. Some AMD Ryzen motherboards can provide what you want, when you use the right PCIe slots and configuration. Mine is an X470 with two (old) AMD GPUs passed to two different VMs, leaving the host headless.

 

[Hiroi]

They are not in the same iommu group, by default the nvidia card is in it's own and the amd one is bundled with some other controller and things like that, so that's why I used ACS override, just so I can have the amd card free from the rest.

 

[Hiroi]

Please check your IOMMU groups without pcie_acs_override and tell us exactly which motherboard you are using. Some AMD Ryzen motherboards can provide what you want, when you use the right PCIe slots and configuration. Mine is an X470 with two (old) AMD GPUs passed to two different VMs, leaving the host headless.

My motherboard is a b550 aorus elite and I answered the iommu groups question in the other response.

 

[leesteken]

Unfortunately, all x16 PCIe slots except the first one on your motherboard are not connected to the CPU but instead to the B550 chipset (and provide only x2 or x1 mode). Sorry to say, but your two GPU passthrough will not work with your current motherboard.
You'll need a AMD motherboard that can split the x16 PCIe lanes from the CPU over two x16 slots (which then both run in x8 mode) to passthrough two GPUs. Or a motherboard that has a secondary x16 in x4 mode that is shared with an M.2 slot (connected to the CPU lanes). You can check for this specification page of the motherboard manufacturer website. I'm sure you can find one that supports your CPU and does support two GPU passthrough. If you want, I can double check for you before you buy it.

 

[Hiroi]

Unfortunately, all x16 PCIe slots except the first one on your motherboard are not connected to the CPU but instead to the B550 chipset (and provide only x2 or x1 mode). Sorry to say, but your two GPU passthrough will not work with your current motherboard.
You'll need a AMD motherboard that can split the x16 PCIe lanes from the CPU over two x16 slots (which then both run in x8 mode) to passthrough two GPUs. Or a motherboard that has a secondary x16 in x4 mode that is shared with an M.2 slot (connected to the CPU lanes). You can check for this specification page of the motherboard manufacturer website. I'm sure you can find one that supports your CPU and does support two GPU passthrough. If you want, I can double check for you before you buy it.

Well that's sad isn't it. I've got some motherboards, but I am not sure about it from their spec sheets.
https://www.asrock.com/mb/AMD/X570 Pro4/index.asp#Specification
https://www.msi.com/Motherboard/B550-A-PRO/Specification
https://www.asrock.com/mb/AMD/B550 Pro4/index.asp#Specification

 

[leesteken]

The MSI B550-A-PRO has a secondary x16 slot (running in x4) connected to the CPU. This will most likely work but you cannot use the fastest M.2 slot.
The ASRock B550 Pro4 looks very similar to the MSI one, but it does not state that the secondary x16 slot (running in x4) is shared. I'm not sure about this one.

Any X570 is the best for passthrough, as even some/most of the devices connected to the chipset are in separate IOMMU groups out of the box. It is the more expensive option and has a fan on the board but does make a big difference if you want to do passthrough of more (and even onboard) devices.
The secondary x16 will also be in x4 mode on the ASRock X570 Pro4 but it will be PCIe 4.0 instead of 3.0, which is useless for the RX540 but is twice as fast for cards that can use it.

EDIT: Please be aware that my advice is an educated guess based on the specifications and I cannot guarantee motherboards without actually using it myself first.

 

[Hiroi]

The MSI B550-A-PRO has a secondary x16 slot (running in x4) connected to the CPU. This will most likely work but you cannot use the fastest M.2 slot.
The ASRock B550 Pro4 looks very similar to the MSI one, but it does not state that the secondary x16 slot (running in x4) is shared. I'm not sure about this one.

Any X570 is the best for passthrough, as even some/most of the devices connected to the chipset are in separate IOMMU groups out of the box. It is the more expensive option and has a fan on the board but does make a big difference if you want to do passthrough of more (and even onboard) devices.
The secondary x16 will also be in x4 mode on the ASRock X570 Pro4 but it will be PCIe 4.0 instead of 3.0, which is useless for the RX540 but is twice as fast for cards that can use it.

EDIT: Please be aware that my advice is an educated guess based on the specifications and I cannot guarantee motherboards without actually using it myself first.

I've just bought a MSI MPG 570 and everything works fine now, I have two vms with two different gpus passed through at the same time. Thank you, I couldn't have fone it without your help!

 

[zackfuchtel]

Hello, sorry for diging in old threads, but I have exact the same problem. Two Nvidia Cards to two VM passtrought, first work, second not. Board is asRockRack x570d4u. IOMEMM groups for eych card seperated (2d:00 2e:00). No ACS Patch. How to configure the bios? Has onboard to be deactivated? Can you help here? Thanks.

 

[zackfuchtel]

Hello, sorry for diging in old threads, but I have exact the same problem. Two Nvidia Cards to two VM passtrought, first work, second not. Board is asRockRack x570d4u. IOMEMM groups for eych card seperated (2d:00 2e:00). No ACS Patch. How to configure the bios? Has onboard to be deactivated? Can you help here? Thanks.

EDIT:
okay, in case someone stumbled over this, my solution: I have to connect a monitor, or in my case, a HDMI dummy plug (ebay 10$) to the onboard GPU. I asume the Bios send something over to the dedicated GPUs when no monitor is detected and this blocks vfio module. Bye.

 

[Kiwi322]

I've just bought a MSI MPG 570 and everything works fine now, I have two vms with two different gpus passed through at the same time. Thank you, I couldn't have fone it without your help!

Hi, I just learnt this by trying 2 gpus on a Asus strix b550-f, which went as well as one would expect. I came across this post while tinkering
https://www.msi.com/Motherboard/MPG-X570-GAMING-PLUS
Is it this link that for the motherboard it works for you?

Thanks for the information provided previously.

 

[AndreasG79]

Hello everyone! Sorry for posting in an old thread but this was what shows up when I googled my problems.
I´ve kind of same problem as zackfuchtel. I am a noob but I know my way around the command line with guidance.
I have 2 nvidia gpus, 1. Nvidia Quadro RTX 4000 2. Nvidia Geforce RTX 2070 super.
I can passthrough Nvidia Geforce RXT 2070super but not my Nvidia Quadro RTX 4000 if I start them amd also if I start just the Nvidia Quadro RTX 4000 alone I get this error in syslog

I could paste all my info here so I needed put everything in textfile


It seems they use the same controller for the Audio with HDMI passthrough and this is same for the usb -c on both gpus
Can this be the problem, as my syslog outputted ? If you need more info about my system please write the commands and I will forward the info you need.
Please can someone help me. It would be much appreciated. Thanks in advance and have a good day all.

 

[leesteken]

I can passthrough Nvidia Geforce RXT 2070super but not my Nvidia Quadro RTX 4000 if I start them amd also if I start just the Nvidia Quadro RTX 4000 alone I get this error in syslog

You need this work-around (since Proxmox 7.2) when doing passthrough of the boot GPU and getting BAR can't reserve mem errors.

 

[AndreasG79]

You need this work-around (since Proxmox 7.2) when doing passthrough of the boot GPU and getting BAR can't reserve mem errors.

Helllo @leesteken Many thanks for the advice! Now my Nvidia Quadro work in my win11 vm. Now my proxmox is perfect! As I understand is this a bug in kernel 5.15? In that case will anyone report this as a bug? Thanks again for your help.