Dual GPU, one passes through, other fails with BAR 1: can't reserve

[he_who_floats_among_us]

As the title says, I'm having a problem getting both of my GPUs to pass through to VM. They're in a single IOMMU group, which I think is fine for my use case (I want them both to go to a single VM).

If I pass in GPU #2, everything works.
If I pass in GPU #1 instead of or in addition to GPU #2, I get some error messages:

- Invalid PCI ROM header signature
- kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0x4020000000-0x402fffffff 64bit pref]

Some possibly relevant information. These are Nvidia RTX 3080. I did vbios dumps for each GPU once upon a time in Windows with GPU-Z. Those dumps were slightly different, even though the cards are identical and share a common set of IDs. I use the romfile option with the VM to load each GPU's rom.

 

[leesteken]

This is typical (with kernel 5.15+) when a GPU is used for systemboot and Proxmox host console prior to starting the VM. Note that passthrough of the boot GPU when you have multiple GPUs is essentially the same as the more common 'single GPU passthrough' issue.
If BOOTFB appears in cat /proc/iomem (after a reboot and before starting the VM), please try this work-around (and there might be more information in that thread). I believe that work-around is necessary but you might need additional step from the 'single GPU passthrough' like early binding the pci-vfio and/or blacklisting drivers.

 

[he_who_floats_among_us]

Thank you for that link. The off-wiki change in that thread was to replace the kernel option video=efifb:off with initcall_blacklist=sysfb_init, which worked perfectly.

 

[zartarr88]

What does ROM BAR 1 error mean ? because making that kernel option didn't work for me. And I've tried plethora of options through the thread mentioned in the work-around by @leesteken . Can someone help thank you .

 

[leesteken]

What does ROM BAR 1 error mean ?

Can you please show the whole error message? Maybe it's not the same?
kernel: vfio-pci 0000:??:??.0: BAR ?: can't reserve [mem 0x???-0x??? 64bit pref] usually means that the GPU was used during boot of the Proxmox host and not properly released by the fb-driver. It will show BOOTFB somewhere in the output of cat /proc/iomem.

because making that kernel option didn't work for me.

initcall_blacklist=sysfb_init should fix the "BAR can't reserve" error. Can you double check with cat /proc/cmdline what the kernel parameter is applied? Can you double check the error message and the iomem (see above)? Or maybe you are running into another issue and more information about your setup is needed for troubleshooting.

 

[zartarr88]

Can you please show the whole error message? Maybe it's not the same?
kernel: vfio-pci 0000:??:??.0: BAR ?: can't reserve [mem 0x???-0x??? 64bit pref] usually means that the GPU was used during boot of the Proxmox host and not properly released by the fb-driver. It will show BOOTFB somewhere in the output of cat /proc/iomem.

initcall_blacklist=sysfb_init should fix the "BAR can't reserve" error. Can you double check with cat /proc/cmdline what the kernel parameter is applied? Can you double check the error message and the iomem (see above)? Or maybe you are running into another issue and more information about your setup is needed for troubleshooting.

I do agree with you that it could potentially be another issue , or a set of compounding issues. The reason I have been adament for 2+ weeks on it is because I have seen at one point or another going through the process that 1 of my GPU VMs will work , and then the other stops. Or vice versa . I just can't have both ... new thread started. Hope to see you there !