[SOLVED] Is PCIe Passthrough supposed to work with Proxmox 7?

[Asano]

I just tried to update a standalone (non-cluster) Proxmox 6.4 with 2 GPUs (1080 and 2080 SUPER) which are both passed through to different VMs. Maybe worth to mention I also pass through some USB controllers directly to those VMs. The mainboard in use is a X570 AORUS Pro. I blacklist GPU drivers so usually after the bootloader there is no more graphic output on the host.

However, after the update to Proxmox 7 the host showed a `EFI stub: loaded initrd from command line option` message after the bootloader. When I tried to start VMs with PCIe passthrough this went away and the screen went black again (so something happened) but it just stayed black. So I didn't even get the logo of the Proxmox boot menu inside the VM.

This was the case for all VMs with PCIe passthrough, Linux as well as Windows guests and for both GPUs, no matter how often I tried. The syslog and dmesg didn't show any to me obvious error message and looked normal. All VMs without any PCIe passthrough started and worked normally.

I haven't looked into it very long and decided to just rollback to 6.4 very soon (luckily I was prepared and now everything works again). Now I'm just wondering is this some kind of known problem or something specific to my system/unknown so far?

Thanks for any hints - since maybe at some point it would be nice if I could update this machine to 7 as well ;-)

 

[jf2021]

Hi ! I've been able to passthrough a PCIe GTX1660 on a fresh install of Proxmox 7 with no problem, as well as many other devices : USB keyboard and mouse, a whole SATA HDD... My MB is a Gigabyte X570

I haven't tried on an upgraded Proxmox but I'd suspect some configuration changes, or device addresses change.

Indeed I noticed some weird things with PCIe card addresses. For example after I physically removed one extra PCI card from the motherboard, the address of the remaining (untouched) GPU had changed in proxmox and therefore it was not working anymore. I had to find the new adress with lspci and update the conf of the VM.

Also I've been having some problems trying to passthrough an old GT640 GPU on Proxmox 7, that works if it's plugged as secondary GPU, but not as first... As you can see on this thread.

Hope this will give you som clues... there are not many answers for that type of questions on this forum unfortunately...

 

[eminempt]

thanks a lot for your insights!! im having problems aswell with that:

I would like to have my Windows VM comming out of my HDMI output on my Graphics card.

Im running PxMox 7 in a ryzen 9 5950x processor, x570 motherboard and a 1650 Nvidia GPU.

Since my CPU does not have Graphics processing unit, i only have one GPU at all. (Nvidia)

so now my question is, would that be the reason why my GPU Passtrought is not working?

 

[Asano]

@eminempt I'm currently not sure if you already had the passthrough working once with Proxmox 6 or if you are trying on a fresh install. Generally speaking I could imagine getting a passthrough to work with one GPU only could be really tricky. For the primary GPU I always needed a ROM file to get passthrough working and here ROM files from the internet (Techpowerup) never worked for me. So I had to extract it myself and that again was only doable while the GPU was *not* the primary which again means at least during that time there should be a second GPU which is the primary. For what it is worth also have a look here: https://forum.proxmox.com/threads/gpu-passthrough-working-for-2-gpu-but-not-for-1.92825/post-411488 but there may be other threads with better info for your case.

Regarding the OT I recently did the in place upgrade from the currently latest 6 to 7 again and now everything worked smooth. Not sure what the actual issue the first time was. I believe in the version back than q35 version pinning was not yet here. However after the upgrade my Windows VMs with passthrough also worked (after some self-adjustments) after setting the q35 version to 6.1...

 

[eminempt]

hey @Asano.
Check that post you just sent me over there, i managed to solve my problem, and replied there! now working perfecly fine with GPU passtrough. just had to give one argument to the Grub and use a .rom file like you mentioned.

And also, i used one from Techpowerup. i Saw a really insighful video on how to make them work !!

here you go:
https://www.youtube.com/watch?v=1IP-h9IKof0

got it working with some tweaks..

also just managed to passtrough the audio devices 1 hour ago.
wasnt working by just adding the device to the vm (throwed me the host into error looping boot state) needed 2 hour TSHing but what was missing was on /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1f82,10de:10fa disable_vga=1
options vfio-pci ids=1022:1487 [audio device]

graphics card was a lot harsher, it was working at some point but w/ Error 43. i noticed that didnt occured while using VGA device (VNC) from the browser.. so i could only assume Proxmox HV aswell was trying to use the GPU aswell. Completly disabled it from the Hypervisor perspective at /etc/default/grub by adding "video=efifbff"

answering your question about PVE version, i tried on 6.4 when i was not as experienced, and now i managed to do it in 7.1.
what i think your problem could've problably been (read somewhere) related w/ the VM .conf file, or any other thing (might be confused at this point lol) whereas new Proxmox 7.1 already has some parameters on the Kernel, and before you would've had to put them on the .conf or other file, i think.

for what its worth, here is the arguments i put on my .conf file. Problably some are not needed anymore (kvm=off, cpu hidden etc) i dont know. its working.

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=sata0
cores: 10
cpu: host,hidden=1


if you need, i've logged all the changes that i did on the Hypervisor! thanks for getting back to me

Marco

 

[eminempt]

hey one question @Asano, just noticed that you have the same motherboard as i have.
Did you managed to make RGB Fusion working to control the motherboards lights? i guess that would not be easy at all?