[SOLVED] GPU Passthrough Issues After Upgrade to 7.2

leesteken

Famous Member
May 31, 2020
2,663
579
118
And there's no way to disable per GPU?
If you have multiple GPUs, you might not need this work-around. It is needed because the passthrough GPU is used during POST/boot, typically because it is the only GPU. Can't you make the system POST/boot with one of the other GPUs (either via the BIOS or by switching the PCIe slots)?
 
Last edited:
Nov 3, 2020
52
19
13
34
I do have a boot GPU specified in BIOS, but this issue affected my system anyways. I have the hookscript setup working well for me, but I have been keeping an eye out for a more "persistent" change. Not sure I want to kill my access to boot logging though.
 

leesteken

Famous Member
May 31, 2020
2,663
579
118
I do have a boot GPU specified in BIOS, but this issue affected my system anyways. I have the hookscript setup working well for me, but I have been keeping an eye out for a more "persistent" change. Not sure I want to kill my access to boot logging though.
That is unexpected. Can you tell me more about which GPUs you have and which one is used for POST/boot/console? Maybe we can find a better/more specific work-around.
 

leesteken

Famous Member
May 31, 2020
2,663
579
118
Host OS GPU (console, post, etc.): ASPEED Graphics Controller (rev41)

Passthrough GPU: NVIDIA GeForce GP106 (GTX1060 6GB)
Did you early bind the GTX1060 to vfio-pci (using ids=... and maybe a soffdep), to prevent it from being used by Proxmox?
 

leesteken

Famous Member
May 31, 2020
2,663
579
118
I don't believe so. I just followed this guide:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
Under Host Configuration that wiki page shows how to add options vfio-pci ids=... to a file in /etc/modprobe.d/ (vfio.conf for example) using the ids of your GTX1060 (use lspci -nn to look them up). If the nouveau driver is (still) loaded for the GPU (check with lspci -nnk), you need to blacklist it or use a softdep.
This should prevent Proxmox (i.e., the Linux kernel) from touching the GPU before starting the VM, which should prevent the need for a work-around for that GPU.
 
Last edited:
Nov 3, 2020
52
19
13
34
Ah, I did end up doing that. Went through my configs and found them as expected. However, with lscpi -nnk are we only concerned with the kernel driver in use, or also the kernel modules? Because as of now I do have the nouveau and nvidia drivers blacklsited and the kernel driver in use is vfio-pci.
 

leesteken

Famous Member
May 31, 2020
2,663
579
118
Ah, I did end up doing that. Went through my configs and found them as expected. However, with lscpi -nnk are we only concerned with the kernel driver in use, or also the kernel modules? Because as of now I do have the nouveau and nvidia drivers blacklsited and the kernel driver in use is vfio-pci.
Yes, the in-use one. If that's just after a reboot of the Proxmox host and before starting the VM, then that's fine. And I don't understand why you need additional work-arounds before starting the VM. (I also hope you did not install the proprietary NVidia drivers on the Proxmox host.)
 
Nov 3, 2020
52
19
13
34
I sure did not install any drivers on the Proxmox host. Only changes I've made to the install are the steps to enable pcie passthrough.

I will verify again next time I can reboot the host, but for what its worth all of this was set up and working before the 5.15 kernel. I only had to start using the hookscript to get passthrough working again under 5.15.
 

leesteken

Famous Member
May 31, 2020
2,663
579
118
I sure did not install any drivers on the Proxmox host. Only changes I've made to the install are the steps to enable pcie passthrough.
Then there should be no driver named nvidia that you need to blacklist.
I will verify again next time I can reboot the host, but for what its worth all of this was set up and working before the 5.15 kernel. I only had to start using the hookscript to get passthrough working again under 5.15.
The most recent 5.15 kernels are better in some respects. With my AMD GPU use for POST?boot messages, I don't even need to blacklist, early bind or work-around the boot display. The amdgpu driver handles it for me. but I did have issues with the early 5.15 kernels. Check cat /proc/iomem for BOOTFB. That's the typical cause of the BAR can't reserve memory errors and usually only for GPU's used for POST/boot/console. I just don't get what is causing the problem that you need the work-around for.

EDIT: Did you run update-initramfs -u and update-grub or proxmox-boot-tool refresh to activate changes to /etc/modprobe.d/. Note that the GTX 1060 should not display anything before starting the VM, otherwise something is wrong.
 
Last edited:
Nov 3, 2020
52
19
13
34
No BOOTFB in /proc/iomem. And I did run the update-initramfs -u and proxmox-boot-tool refresh. As stated, it worked fine before the 5.15 branch. I am still on 5.15.35-6, so maybe I need to update again and try removing the hookscript and see what happens.
 
Nov 3, 2020
52
19
13
34
As an update:

Updated to the 5.15.39-1 kernel this morning and disabled the hookscript. Upon reboot the VM started without issue, and I verified passthrough was behaving as expected. So it looks like at some point a kernel update resolved the issue causing the framebuffer to improperly reserve the secondary GPU.
 

marcosscriven

Member
Mar 6, 2021
96
7
8
As an update:

Updated to the 5.15.39-1 kernel this morning and disabled the hookscript. Upon reboot the VM started without issue, and I verified passthrough was behaving as expected. So it looks like at some point a kernel update resolved the issue causing the framebuffer to improperly reserve the secondary GPU.
I found this too - and I'm really itching to understand why.
 

Chan Tai Man

Active Member
Nov 11, 2016
11
0
26
56
I found this too - and I'm really itching to understand why.
I use below sources.list and command 'apt-get upgarde'

deb http://ftp.hk.debian.org/debian bullseye main contrib
deb http://ftp.hk.debian.org/debian bullseye-updates main contrib
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# security updates
deb http://security.debian.org bullseye-security main contrib

1.png

I cannot upgrade to 5.15.39-1 kernel.

Please tell how to upgarde the kernel.

Thanks
 
Last edited:

leesteken

Famous Member
May 31, 2020
2,663
579
118
I use below sources.list and command 'apt-get upgarde'

deb http://ftp.hk.debian.org/debian bullseye main contrib
deb http://ftp.hk.debian.org/debian bullseye-updates main contrib
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# security updates
deb http://security.debian.org bullseye-security main contrib

I cannot upgrade to 5.15.39-1 kernel.

Please tell how to upgarde the kernel.

Thanks
Never run apt-get upgrade. Run apt-get update and then apt-get dist-upgrade. Click the manual for more information.
 

Chan Tai Man

Active Member
Nov 11, 2016
11
0
26
56
Never run apt-get upgrade. Run apt-get update and then apt-get dist-upgrade. Click the manual for more information.
Thanks

After I upgrade to 5.15.39-1 kernel, I still cannot solve the passthrough GPU issue. The config worked in 7.1

kvm: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=1,romfile=/root/vbios.rom: Failed to mmap 0000:00:02.0 BAR 2. Performance may be slow
kvm: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=1,romfile=/root/vbios.rom: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
TASK OK
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!