Can't get GPU passthrough working again

M

MysticPrysm

New Member
Aug 13, 2022
18
0
1
So, I'm going to paste what I posted on reddit, and maybe build on that.

So, I've reinstalled proxmox. I had the passthrough working before using various guides on the internet either from craftcomputing, or the beginners guide commonly found on the homelab subreddit. But as I've read recently, kernel 5.15 broke it for loads of people. Now, I'm basically full on noob. I'm not great with linux, but I'm great at following directions. So, if someone could explain to me what I'm missing, as if I was a toddler, I'd appreciate it. lol.

Trying to passthrough to windows 10/11

So far, I've basically followed the whole guide, but replaced the grub section about efifb with

GRUB_CMDLINE_LINUX_DEFAULT="quiet initcall_blacklist=sysfb_init"

I'm running a ryzen 7 3700x, a 1070ti, an x470 board, aaand, i also have umm, some R9 card, I don't remember the exact model as I got it used. I think like a 270.

Basically, any of the fixes I've tried, it results in the VM not wanting to start, it locks up proxmox, and I have to wait to access it again or do a hard reset.

I've tried the various fixes I've found regarding efifb and otherwise, but nothing has worked.

When I hit start on the VM, proxmox locks up, and the VM doesn't actually start. I either have to wait a few minutes or hard reset to gain access to the web interface again.

If I remove the PCI device, the VM will boot right up.
 
MysticPrysm said:
That says it's for amd workarounds, but it should work with Nvidia?
Click to expand...
No, only for AMD. I thought you wanted to passthrough the AMD GPU, sorry.
initcall_blacklist=sysfb_init is only needed the BOOTFB occurs in cat /proc/iomem for the GPU that is used for boot and Proxmox host console.
Can you show your iOMMU groups with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? Maybe Proxmox locks up because the NVidia GPU is not isolated in its own group.
EDIT: Maybe you used pcie_acs_override before? I don't expect you to need it with an X470, if the GPU is in the first or second x16 PCIe slot.
 
Last edited:
leesteken said:
No, only for AMD. I thought you wanted to passthrough the AMD GPU, sorry.
initcall_blacklist=sysfb_init is only needed the BOOTFB occurs in cat /proc/iomem for the GPU that is used for boot and Proxmox host console.
Can you show your iOMMU groups with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? Maybe Proxmox locks up because the NVidia GPU is not isolated in its own group.
EDIT: Maybe you used pcie_acs_override before? I don't expect you to need it with an X470, if the GPU is in the first or second x16 PCIe slot.
Click to expand...
root@proxmox:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 11 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 12 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 13 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 13 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 14 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 14 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 14 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 14 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 14 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 14 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 14 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 14 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 15 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU group 16 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
IOMMU group 17 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU group 18 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU group 19 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 21 20:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 22 20:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 23 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 24 20:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU group 25 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 26 26:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU group 27 27:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] [10de:1b82] (rev a1)
IOMMU group 28 27:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU group 29 28:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
IOMMU group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 30 28:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
IOMMU group 31 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 32 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 33 2a:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 34 2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 35 2a:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 9 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
 
It looks like you are already using pcie_acs_override because no X470 motherboard has such small groups, so I can't tell whether it's an IOMMU group problem.
It does look like the system will boot with the NVidia GPU. Can you try swapping the GPUs and make it boot with the AMD GPU? Maybe sure to change and double check the PCI ID's in the VM configuration(s).
EDIT: Also, don't give too much memory to a VM with passthrough because all of it must be pinned into actual host memory. This can cause issues if you run low (over 80%) on memory for the host.
 
Last edited:
The only reason I haven't tried that is because technically, if I use anything other than the first four slot, won't the card technically run slower even through a vm?
 
Can you please share the make and model of your X470 motherboard? Usually they have two x16 PCIe slots, with both work at x8 when they are both used. Which slots are you using at the moment?
 
leesteken said:
Can you please share the make and model of your X470 motherboard? Usually they have two x16 PCIe slots, with both work at x8 when they are both used. Which slots are you using at the moment?
Click to expand...
MSI Gaming Plus Max x470

3 slots. Nvidia in first one. And in second.
 
Sorry, I'm stumped. Is there anything in the Syslog or journalctl from around a time that you started the VM (until the forced reboot) that might give a clue?

PS: If there is anything in the fourth slot, the first slot with the NVidia GPU will run in x8 mode according to the detailed specifications.
 
I don't know how to get the syslog either lol. See I'm good at doing things, but I just don't know how to do most things on Linux lol.
 
Either have a look at the Proxmox web GUI, go to the node and then System, Syslog. Or login to the Proxmox host console (or SSH or the shell in the Proxmox web GUI) and run journalctl.
 
MysticPrysm said:
BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet pcie_acs_override=downstream,multifunction
Click to expand...
initcall_blacklist=sysfb_init is missing. Maybe that's my fault for being unclear and confusing, but you really need that work-around for passthrough of a NVidia GPU that is also used during boot and for the Proxmox console (with kernel 5.15 or higher). Change /etc/default/grub, run update-grub, reboot and check again.
 
Last edited:
I believe I had it at one point with same results. Of course all of my configs are a little modified from original guide anyway. But I'll try this and report back in a minunte.
 
MysticPrysm said:
I believe I had it at one point with same results. Of course all of my configs are a little modified from original guide anyway. But I'll try this and report back in a minunte.
Click to expand...
Maybe you had the same results because of another issue, but the work-around is necessary (and reboot after update-grub).
 
like this correct?

Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init"
 
You must log in or register to reply here.
Top Bottom