[SOLVED] Command line from /etc/default/grub not applied and PCI passthrough not working

[Asano]

I'm playing around with Proxmox 6 and PCI/GPU passthrough in a desktop PC with two graphic cards. I basically followed the Proxmox guides and some other tutorials out there. The setup also already works perfectly for both GPUs with near native performance in the VMs but unfortunately only as long as the GPU is not set as primary GPU. The GPU I select as primary in my motherboard's BIOS will not work and therefore only one GPU is working as passthrough GPU at a time which is not what I want.

So I'm kind of certain this is due to some conflict caused by the host which also tries or tried to use the GPU. I've read quite a lot about the topic and tried nearly all possible combinations of video=vesafb:off,efifb:off , video=vesafb:off video=efifb:off and so an as grub GRUB_CMDLINE_LINUX_DEFAULT and also GRUB_CMDLINE_LINUX addition since I'm guessing this would/could fix the problem.

However, no matter what I do in my /etc/default/grub file it seems it has no effect at all. Obviously I also run update-grub afterwards (and I even tried update-initramfs -u and pve-efiboot-tool refresh).
Currently I have GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs video=vesafb:off video=efifb:off"

But as always cat /proc/cmdline ends up with

initrd=\EFI\proxmox\5.0.21-5-pve\initrd.img-5.0.21-5-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

I'm not sure if the grub file change should reflect in /proc/cmdline but I would have guessed it. Also dmesg | grep "efifb" shows the following which I think indicates that other than intended efifb is actually used (note that 0000:0c:00.0 is actually the GPU currently selected as primary):

[    0.206549] pci 0000:0c:00.0: BAR 3: assigned to efifb
[    0.633202] efifb: probing for efifb
[    0.633215] efifb: framebuffer at 0xf1000000, using 3072k, total 3072k
[    0.633217] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    0.633218] efifb: scrolling: redraw
[    0.633219] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0

I also see the system output on the connected monitor until about [ 5.000000]. Than it freezes also the system continues to boot correctly (so I guess after about 5 seconds the vfio config from etc/modprobe.d takes effect, but that's also just a guess...).

So is there a way to get Proxmox to respect the /etc/default/grub setting and actually disable the efifb? (Or am I facing a entirely different problem for which maybe is a different workaround?)

Any tips are greatly appreciated!

 

[dcsapak]

if you run zfs with uefi, we do not use grub for booting and the steps to adapt the kernel command line are different, see https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline

 

[Asano]

if you run zfs with uefi, we do not use grub for booting and the steps to adapt the kernel command line are different, see https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline

Now I feel silly Thanks so much for the info - that was effective!

However, GPU passthrough still doesn't work for the primary GPU. Now without any error (as far as I can see) but a black screen after sysboot console which switches to no signal when the VM tries to claim it. However now that I can play around with the command line I can try more stuff and would open a new thread shouldn't I be able to fix it.

And another side note. It looks like the command line options from https://pve.proxmox.com/wiki/Pci_passthrough and https://pve.proxmox.com/wiki/PCI(e)_Passthrough aren't required anymore as well since obviously I added them to grub file before as well and thus they were never effective on my system. Still GPU passthrough worked flawless for the non-primary GPU and Interrupt remapping was always enabled nevertheless.

 

[dcsapak]

However, GPU passthrough still doesn't work for the primary GPU. Now without any error (as far as I can see) but a black screen after sysboot console which switches to no signal when the VM tries to claim it. However now that I can play around with the command line I can try more stuff and would open a new thread shouldn't I be able to fix it.

yeah passing theprimary gpu through can be a pain (depending on the hardware)

And another side note. It looks like the command line options from https://pve.proxmox.com/wiki/Pci_passthrough and https://pve.proxmox.com/wiki/PCI(e)_Passthrough aren't required anymore as well since obviously I added them to grub file before as well and thus they were never effective on my system. Still GPU passthrough worked flawless for the non-primary GPU and Interrupt remapping was always enabled nevertheless.

there are apparently some circumstances where the iommu is activated by default, but this is not the case for all platforms, so we will leave that portion in the docs (for now)

 

[Asano]

yeah passing theprimary gpu through can be a pain (depending on the hardware)

Seems like it but now everything works for me. And since I had to do some things a little different from what I found in the Proxmox guides and forum I thought I post it real quick:

1. Set up like described in the Promox guides posted above
2. In the VMs .conf files I had to differ a little bit from the guides. Based on Proxmox forum tips I currently have the following lines different from Proxmox GUI config and guides:

args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off,hv_vendor_id=intel'
machine: pc-q35-3.1
cpu: host,hidden=1

(At this point non-primary GPU passthrough was already working very well for me.)
3. Add command line options; mine now looks like this root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on rd.driver.pre=vfio-pci video=vesafb:off video=efifb:off but I think actually root=ZFS=rpool/ROOT/pve-1 boot=zfs video=efifb:off would already work.
4. To get the primary GPU to work I also needed to pass its rom file. However extracting it just like described in https://pve.proxmox.com/wiki/Pci_passthrough#romfile did not work and also the image from https://techpowerup.com/vgabios/ did not work (also my exact card was listed).
The tweak which did the trick was posted in the unRAID forum by planetwilson and basically is to config the GPU as non-primary AND boot up a windows VM with it, just to sort of "initialize" it's BIOS. After checking that the VM works like it should including the passed through GPU just shut it down again. So all steps:

# Ensure the GPU you want to extract the BIOS from is not the primary
# Boot up Windows VM with the GPU passed through and graphic driver installed and working
# Shut down the Windows VM
echo "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
cd /sys/bus/pci/devices/0000:0b:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/MSI.GT1030-extracted.rom
echo 0 > rom
echo "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

And now just pass the rom file in the VM's config file: hostpci0: 0b:00,pcie=1,x-vga=1,romfile=MSI.GT1030-extracted.rom

I have no idea why booting up a windows VM with the card is important but also the rom file size did differ greatly so it really seams to make a difference. This are my rom files (and only the MSI.GT1030-extracted.rom worked, which is the one from after booting up the VM):

/usr/share/kvm# ls -l |grep MSI
-rw-r--r-- 1 root root   57856 Nov 28 01:51 MSI.GT1030-extracted-old-method.rom
-rw-r--r-- 1 root root  128000 Dec  2 19:56 MSI.GT1030-extracted.rom
-rwxr-xr-x 1 root root  258560 Oct 23 02:14 MSI.GT1030-techpowerup-download.rom

 

[Old-Skoolz]

I would like to thank the both of you for your input on this issue.

I just installed Proxmox for the first time last weekend and was flooded by this error: "AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0011..[/COLOR]." I ran across a youtuber that said he uses AMD graphics cards with AMD chipset mainboards and has less issues in Linux. So I replaced my Nvidia graphics card with an AMD graphics card and that issue went away.

However, I am now getting flooded with messages "a monitor was probed but no | invalid EDID."

Spent half the day working on finding the solution and was beating on grub all morning. I found this thread and both of you guys saved the rest of the day. I am installed with ZFS Raid1.

Anso pretty much handed me the info to that I needed to add to the cmdline, and Dcsapak provided the link.

The only error in my kernel log now is [drm:radeon_init [radeon]] *ERROR* No UMS support in radeon module!
This only appears once, not spammed like my other errors. Found a thread that says this is because of nomodeset, but if I remove that, my EDID issue returns. I'm hoping I can get by with this one error.

Now I just gotta figure out how to use this Proxmox and decide if I want to keep it. If so, I will definitely buy a subscription, as I believe in paying for what your get.

I am wondering why you would need to passthrough your video card? I would only be installing text based servers and can't think of any reason I would need any graphics abilities.