proxmox 6.2 breaks gpu passthru

[DomF]

Hi,

I've just updated from proxmox 6.1 to 6.2 and now the GPU passthru fails to function on my existing ubuntu guest VM. At first I though maybe the pci host device numbering may have changed after the update but after checking lspci it is the same as in my VM conf file.

Any ideas?

 

[Joris L.]

oh, you mean my fresh install of proxmox 6.2 will simply not work for GPU pass throug?

 

[DomF]

I haven't tried a fresh install but upgraded from 6.1 to 6.2 by installing updates. Something in proxmox 6.2 has changed that stops a previously working VM config setup for GPU passthru now only comes up with a blank screen.

 

[Joris L.]

the VM or the HOST ?

After much trial and error i found efifb=off is the only setting which appears to work for my setup, still, no way for me to test if the admgpu is actually working, it is and it is not afaik.

... oh wait, i get what you mean ... after installing Radeon drivers in the VM after the reboot there is now just a black screen for the Microsoft Windows VM

 

[DomF]

From reading around I have my grub setup as this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 video=efifbff"

but still I have no GPU passthrough functioning.

my VM conf file is this:

#ubuntu
#
#ip%3A 196.168.2.176
balloon: 0
bios: ovmf
bootdisk: virtio0
cores: 8
efidisk0: zfsimage:vm-304-disk-1,size=1M
hostpci0: 08:00,pcie=1,romfile=GTX1050TiMod.rom,x-vga=on
ide2: zfsiso:iso/ubuntu-18.04-desktop-amd64.iso,media=cdrom
machine: q35
memory: 8096
name: ubuntu-vm-304
net0: virtio=AA:E3:955:EB:21,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=1970d99b-30b5-453d-83cb-67a74b79c67a
sockets: 1
usb0: host=04f2:0402,usb3=1
usb1: host=046d:c058,usb3=1
usb2: host=13fd:3940,usb3=1
virtio0: zfsimage:vm-304-disk-0,cache=writeback,discard=on,size=40G
vmgenid: 2753a4ae-0e8a-462e-ba87-457bd865ea94

 

[DomF]

dmesg reports:

[93022.808178] vfio-pci 0000:08:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[93022.810196] vfio-pci 0000:08:00.0: No more image in the PCI ROM

 

[Joris L.]

try efifb=off instead and also inject iommu=pt

 

[DomF]

I've tried that now and still no success. Not sure what else to do. Any other ideas?

 

[Joris L.]

Well. I feel like i myself am running in circles really.

I think i am close now since i no longer get any BAR: ... mem notifications

I will share my config for you to compare, try, evaluate. This config works well but the VM get stuck with a black screen. I assume i will have to try out video=vesafb: off,efifb: off and video=efifb: off,vesafb: off as well

iommu=pt amd_iommu=forced_isolation noats nobar root=ZFS=rpool/ROOT/pve-1 boot=zfs video=efifb ff,vesafb ff hugepagesz=2M hugepages=1375 default_hugepagesz=1G hugepagesz=1G hugepages=8 text

/etc/modprobe.d/passthrough.conf (where nnnn replace the real id i have configured)

blacklist amdgpu​

options vfio-pci ids=nnnn:nnnn,nnnn:nnnn disable_vga=1​

option kvm ignor_ms​

/etc/modules

vfio​

vfio_iommu_type1​

vfio_pci​

vfio_virqfd​

​

aufs​

overlay​

The problem is not getting the PCI-E passthrough to work, it is for it to remain stable.

After a while i get the dreaded vfio_ecap_init: hiding ecap 0xffff.... while initially all is well.
Eventually, after hundreds and thousands of vfio_ecap... messages i get Refused to change power state, currently in D3.

NOTE: after the final reboot i now appear to have working pci-e GPU pass through with video=vesafb: off,efifb: off and the BIOS set to UEFI only. Some suggest to disable CSM is you have that in BIOS.

dmesg | grep BAR
now shows

.... BAR 0: assigned to efifb

which i assume is a good thing since i no longer get flooded with other BAR .. .notifciations

 

[TorqueWrench]

This has been a well-documented problem with the upgrade to 6.2 (and additionally seems to also be caused by Windows/driver updates).

To fix it, you need to enable MSI support on the GPU: https://forums.engineerworkshop.com...n-windows-10-vm-in-proxmox/207?u=torquewrench

1) Turn on display in the VM config (VGA)
2) Get into safe mode
3) Regedit to enable MSI mode.

 

[Joris L.]

This has been a well-documented problem with the upgrade to 6.2 (and additionally seems to also be caused by Windows/driver updates).

To fix it, you need to enable MSI support on the GPU: https://forums.engineerworkshop.com...n-windows-10-vm-in-proxmox/207?u=torquewrench

1) Turn on display in the VM config (VGA)
2) Get into safe mode
3) Regedit to enable MSI mode.

will check, not using Microsoft Windows for all use cases

 

[TorqueWrench]

will check, not using Microsoft Windows for all use cases

It's a good thing to enable regardless, even if it sounds like this may not be your specific problem, in general MSI will give you much better performance.

 

[Joris L.]

It's a good thing to enable regardless, even if it sounds like this may not be your specific problem, in general MSI will give you much better performance.

Just checked, on by default for Linux.
Will use the guide for the MS VM, thanks.

 

[DomF]

The issue I'm having is with my ubuntu VM with the GPU not passing through to it.

 

[Joris L.]

The issue I'm having is with my ubuntu VM with the GPU not passing through to it.

I had the same issues, all seem fixed now, try settings i shared and what is here below.

There is actually a vfio-pci module parameter disable_idle_d3 which can be set to ON

modinfo vfio-pci

edit the file in /etc/modprobe.d where you have configured vfio-pci and append
disable_idle_d3=1

reboot, then check the dmesg output, now you should see normal mention of d3

dmesg | grep -i d3

 

[Joris L.]

DAMNED, no longer D3 messages but once more the dreaded vfio_ecap_init: hiding ecap

 

[TorqueWrench]

Just checked, on by default for Linux.
Will use the guide for the MS VM, thanks.

Great, that solves that mystery. I have a GPU passed through to a Linux VM, but I haven't spun that VM up since my data science class.

I need to update that guide; if you can get into safe mode, you can do the regedit from there and skip the whole gimping the GPU thing to make the VM boot.

DAMNED, no longer D3 messages but once more the dreaded vfio_ecap_init: hiding ecap

I saw this, video=efifb: off, in your post further up. Do you really have a space there?

 

[Joris L.]

Great, that solves that mystery. I have a GPU passed through to a Linux VM, but I haven't spun that VM up since my data science class.

I need to update that guide; if you can get into safe mode, you can do the regedit from there and skip the whole gimping the GPU thing to make the VM boot.



I saw this, video=efifb: off, in your post further up. Do you really have a space there?

Not really, i saw no other way to avoid video=efifb: off look like video=efifbff

 

[TorqueWrench]

What GPU is this?

I vaguely remember hearing problems with Vega specifically:

https://forum.level1techs.com/t/vega-passthrough-on-unraid-completely-lost/141772/2

But also RX580:

https://gitlab.freedesktop.org/drm/amd/-/issues/346

This might be fixed (or at least a workaround) if this in fact the problem I'm thinking of:

https://www.reddit.com/r/VFIO/comments/dxhdxc/current_state_of_the_amd_reset_bug/

https://forum.level1techs.com/t/navi-reset-kernel-patch/147547

Sorry I can't be of better assistance as my Proxmox server is both Intel and Nvidia. Your problem is of special interest to me though because I most likely will be going over to AMD in my next production build.

-TorqueWrench

 

[DomF]

Hi,

Have you tried spinning up your linux VM with gpu passthru? Does it start up?


I'm running Ryzen 1700 with Nvidia 1050ti GPU and ROM image.