Proxmox 9.1.9 -> 9.2.4 broke VFIO passthrough to Windows 11 guest

D

djx

Active Member
Aug 11, 2020
16
2
43
38
Last night I upgraded my Proxmox from 9.1.9 to 9.2.4 and the PCI passthrough stopped working. I can still see the ProxMox boot screen on the passed through video card, but once it hands off to the OS it goes blank.

I have a Windows 11 guest that I pass a Radeon 6650 XT through to using this vfio config:
options vfio-pci ids=1002:73ef,1002:ab28 disable_vga=1
Click to expand...

I made no other changes, but suddenly the guest is giving the dreaded 43 error.
I have done multiple full shut down / start ups on the host to ensure the GPU power has been reset.

I have confirmed on the host:
Kernel driver in use: vfio-pci
Click to expand...

I have used a separate guest (CachyOS) to confirm that I can pass through the device and display and I can see output on a physical monitor connected to the passed through device.

I have tried pinning to the old kernel that worked before `7.0.0-3-pve` as well as specifying the old Q35 machine.

I'm at a loss for what else to try next, or how else to roll things back to allow this one VM to work again.

Any help would be greatly appreciated!

Here is the current config:
Code: 
agent: 1
balloon: 0
bios: ovmf
boot: order=ide2;ide0;net0
cores: 24
cpu: host,flags=+pdpe1gb
hostpci0: 0000:0d:00,pcie=1
ide0: disks:109/vm-109-disk-0.qcow2,backup=0,size=1000G,ssd=1
ide2: none,media=cdrom
machine: pc-q35-10.1
memory: 32768
meta: creation-qemu=10.1.2,ctime=1764572762
name: dm3
net0: virtio=...,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=...
sockets: 1
unused0: disks:109/vm-109-disk-1.qcow2
unused1: disks:109/vm-109-disk-2.qcow2
vga: none
vmgenid: ...
 
I spent more time banging my head on this today.

What didn't work:
- Rolling QEMU back to 10
- Hard-coding the same OVMF version that worked before

What ultimately did work: Disabling resizable BAR in BIOS. This worked fine for years, but suddenly did not.

I'm not sure exactly which component updated that caused this to stop working for Windows guests, but after changing the BIOS I could see the memory regions reported to Proxmox updated.

Before:
Region 0: Memory at 7c00000000 (64-bit, prefetchable) [size=8G]
Region 2: Memory at 7e00000000 (64-bit, prefetchable) [size=256M]
BAR 0: current size: 8GB, supported: 256MB … 8GB
Click to expand...

After:
Region 0: Memory at 7fd0000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at 7fe0000000 (64-bit, prefetchable) [size=2M]
BAR 0: current size: 256MB
Click to expand...
 
maybe a kernel issue? if you have the time (and are willing ofc ;) ) could you reenable resizable bar in the bios and try with an older kernel?
 
dcsapak said:
maybe a kernel issue? if you have the time (and are willing ofc ;) ) could you reenable resizable bar in the bios and try with an older kernel?
Click to expand...
Changing the BAR setting to off was the last thing I did, so I can confirm that it being ON (the previous working state) did not work with the following older kernels after the upgrade:
6.14.11-9-pve
7.0.0-3-pve
 
ok, if neither qemu downgrade nor kernel downgrade changes anything, then i don't really have an idea what might have caused this change. There are not that many components involved, and you ruled out the main ones.

Did something else change maybe? windows update? driver update (in the guest) ? bios update?
 
dcsapak said:
ok, if neither qemu downgrade nor kernel downgrade changes anything, then i don't really have an idea what might have caused this change. There are not that many components involved, and you ruled out the main ones.

Did something else change maybe? windows update? driver update (in the guest) ? bios update?
Click to expand...
No BIOS updates, older driver versions didn't work. It's possible there was a windows update, but it's the IoT edition so it updates less aggressively than most 11 installs. If it is on the guest side, I guess it's something that one of the upstream packages will need to fix eventually.

I guess I will keep it this way for now and see if it changes on the next Proxmox update. Thanks for taking a look. :)
 
Was the original version verified and restored?

If it’s actually the result of an intentional change, we’ll just end up waiting indefinitely for it to appear.

Code: 
LC_ALL=C  journalctl --grep "build@proxmox"
tail -n 100 /var/log/apt/history.log
pveversion -v

By the way, I’m wondering if the following is still necessary. Although the GPU is different, my system is running fine without it.

Since the switch from amdgpu to vfio-pci happens seamlessly when the virtual machine starts up, I don’t think there’s a need for early binding.

If I remove this, will the amdgpu driver be able to assign ReBar? If so, will the virtual machine still function properly when it starts up?

Code: 
options vfio-pci ids=1002:73ef,1002:ab28 disable_vga=1

Since the GPUs are different, it might not be the same, but with the latest kernel, the rx9060xt is set to vfio-pci even without that, and Rebar is running without any issues.
Wouldn't it be better for the amdgpu driver to allocate it at startup?

Code: 
7 16 05:08:58 pve2 kernel: amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GP17.VGA_.ATPX handle
7 16 05:08:58 pve2 kernel: amdgpu: ATPX version 1, functions 0x00000000
7 16 05:08:58 pve2 kernel: amdgpu: Virtual CRAT table created for CPU
7 16 05:08:58 pve2 kernel: amdgpu: Topology: Add CPU node
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: enabling device (0006 -> 0007)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: initializing kernel modesetting (IP DISCOVERY 0x1002:0x7590 0x1EAE:0x8601 0xC0).
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: register mmio base: 0xDF000000
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: register mmio size: 524288
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 0 <common_v1_0_0> (soc24_common)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 1 <gmc_v12_0_0> (gmc_v12_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 2 <ih_v7_0_0> (ih_v7_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 3 <psp_v14_0_0> (psp)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 4 <smu_v14_0_0> (smu)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 6 <gfx_v12_0_0> (gfx_v12_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 7 <sdma_v7_0_0> (sdma_v7_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 8 <vcn_v5_0_0> (vcn_v5_0_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 9 <jpeg_v5_0_0> (jpeg_v5_0_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: detected ip block number 10 <mes_v12_0_0> (mes_v12_0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: Fetched VBIOS from VFCT
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] ATOM BIOS: 113-44TC6SHB1-P04
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: Trusted Memory Zone (TMZ) feature not supported
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: VRAM: 16304M 0x0000008000000000 - 0x00000083FAFFFFFF (16304M used)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] Detected VRAM RAM=16304M, BAR=16384M
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] RAM width 128bits GDDR6
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0:  16304M of VRAM memory ready
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0:  40003M of GTT memory ready.
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] GART: num cpu pages 131072, num gpu pages 131072
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] PCIE GART of 512M enabled (table at 0x00000083DAB00000).
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] Loading DMUB firmware via PSP: version=0x0A003A00
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.12 DEC: 9 VEP: 0 Revision: 26
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: MES: vmid_mask_mmhub 0x0000ff00, vmid_mask_gfxhub 0x0000ff00
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: MES: gfx_hqd_mask 0x000000fe, compute_hqd_mask 0x0000000c, sdma_hqd_mask 0x000000fc
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: RAS: optional ras ta ucode is not available
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00664600 (102.70.0)
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: SMU is initialized successfully!
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] Display Core v3.2.369 initialized on DCN 4.0.1
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] DP-HDMI FRL PCON supported
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] DMUB hardware initialized: version=0x0A003A00
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] DP-1: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] DP-2: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] HDMI-A-1: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
7 16 05:08:58 pve2 kernel: kfd kfd: Allocated 3969056 bytes on gart
7 16 05:08:58 pve2 kernel: kfd kfd: Total number of KFD nodes to be created: 1
7 16 05:08:58 pve2 kernel: amdgpu: Virtual CRAT table created for GPU
7 16 05:08:58 pve2 kernel: amdgpu: Topology: Add GPU node [0x1002:0x7590]
7 16 05:08:58 pve2 kernel: kfd kfd: added device 1002:7590
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: SE 2, SH per SE 2, CU per SH 8, active_cu_number 32
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
7 16 05:08:58 pve2 kernel: amdgpu: HMM registered 16304MB device memory
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: Using BACO for runtime pm
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] Registered 4 planes with drm panic
7 16 05:08:58 pve2 kernel: [drm] Initialized amdgpu 3.64.0 for 0000:03:00.0 on minor 0
7 16 05:08:58 pve2 kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes

7 16 05:51:02 pve2 pvedaemon[21325]: start VM 724: UPID:pve2:0000534D:0000A8F0:6A57F2B6:qmstart:724:root@pam:
7 16 05:51:02 pve2 pvedaemon[20384]: <root@pam> starting task UPID:pve2:0000534D:0000A8F0:6A57F2B6:qmstart:724:root@pam:
7 16 05:51:02 pve2 systemd[1]: Stopping lightdm.service - Light Display Manager...
7 16 05:51:02 pve2 systemd[1]: session-c1.scope: Killing process 2168 (lightdm) with signal SIGTERM.
7 16 05:51:02 pve2 systemd[1]: Stopping session-c1.scope - Session c1 of User lightdm...
7 16 05:51:02 pve2 lightdm[2168]: pam_unix(lightdm-greeter:session): session closed for user lightdm
7 16 05:51:02 pve2 lightdm[2168]: pam_unix(lightdm-greeter:session): session closed for user lightdm
7 16 05:51:02 pve2 lightdm[2168]: pam_systemd(lightdm-greeter:session): Failed to release session: Transport endpoint is not connected
7 16 05:51:02 pve2 systemd[1]: session-c1.scope: Deactivated successfully.
7 16 05:51:02 pve2 systemd[1]: Stopped session-c1.scope - Session c1 of User lightdm.
7 16 05:51:02 pve2 systemd[1]: session-c1.scope: Consumed 8.866s CPU time, 57.3M memory peak.
7 16 05:51:02 pve2 systemd-logind[1301]: Removed session c1.
7 16 05:51:02 pve2 systemd[1]: lightdm.service: Deactivated successfully.
7 16 05:51:02 pve2 systemd[1]: Stopped lightdm.service - Light Display Manager.
7 16 05:51:02 pve2 systemd[1]: lightdm.service: Consumed 2.058s CPU time, 97M memory peak.
7 16 05:51:07 pve2 kernel: amdgpu 0000:03:00.0: finishing device.
7 16 05:51:07 pve2 kernel: amdgpu 0000:03:00.0:  ttm finalized
7 16 05:51:09 pve2 kernel: pci 0000:03:00.0: BAR 0 [mem 0xf800000000-0xfbffffffff 64bit pref]: releasing
7 16 05:51:09 pve2 kernel: pci 0000:03:00.0: BAR 2 [mem 0xfc00000000-0xfc0fffffff 64bit pref]: releasing
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0: bridge window [mem 0xf800000000-0xfc0fffffff 64bit pref]: releasing
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0: bridge window [mem 0xf800000000-0xfc0fffffff 64bit pref]: releasing
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1: bridge window [mem 0xf800000000-0xfc0fffffff 64bit pref]: releasing
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1: bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]: assigned
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0: bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]: assigned
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0: bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]: assigned
7 16 05:51:09 pve2 kernel: pci 0000:03:00.0: BAR 0 [mem 0x1c00000000-0x1fffffffff 64bit pref]: assigned
7 16 05:51:09 pve2 kernel: pci 0000:03:00.0: BAR 2 [mem 0x2000000000-0x200fffffff 64bit pref]: assigned
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1: PCI bridge to [bus 01-03]
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1:   bridge window [io  0xf000-0xffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1:   bridge window [mem 0xdf000000-0xdf1fffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:00:01.1:   bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0: PCI bridge to [bus 02-03]
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0:   bridge window [io  0xf000-0xffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0:   bridge window [mem 0xdf000000-0xdf0fffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:01:00.0:   bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0: PCI bridge to [bus 03]
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0:   bridge window [io  0xf000-0xffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0:   bridge window [mem 0xdf000000-0xdf0fffff]
7 16 05:51:09 pve2 kernel: pcieport 0000:02:00.0:   bridge window [mem 0x1c00000000-0x200fffffff 64bit pref]
7 16 05:51:11 pve2 kernel: vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=io+mem:owns=none
7 16 05:51:11 pve2 kernel: vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
7 16 05:51:11 pve2 kernel: vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
7 16 05:51:11 pve2 kernel: vfio-pci 0000:03:00.0: resetting
7 16 05:51:11 pve2 kernel: vfio-pci 0000:03:00.0: reset done
7 16 05:51:11 pve2 pvedaemon[21325]: error writing '1' to '/sys/bus/pci/devices/0000:03:00.0/reset': Inappropriate ioctl for device
7 16 05:51:11 pve2 pvedaemon[21325]: failed to reset PCI device '0000:03:00.0', but trying to continue as not all devices need a reset
7 16 05:51:11 pve2 kernel: snd_hda_intel 0000:03:00.1: GPU sound probed, but not operational: please add a quirk to driver_denylist
7 16 05:51:12 pve2 kernel: vfio-pci 0000:06:00.0: resetting
7 16 05:51:12 pve2 kernel: vfio-pci 0000:06:00.0: reset done

7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.0: resetting
7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.0: reset done
7 16 05:51:14 pve2 kernel: vfio-pci 0000:06:00.0: resetting
7 16 05:51:14 pve2 kernel: vfio-pci 0000:06:00.0: reset done
7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.0: resetting
7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.1: resetting
7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.0: reset done
7 16 05:51:14 pve2 kernel: vfio-pci 0000:03:00.1: reset done
7 16 05:51:14 pve2 kernel: vfio-pci 0000:06:00.0: resetting
7 16 05:51:14 pve2 kernel: vfio-pci 0000:06:00.0: reset done
7 16 05:51:14 pve2 pvedaemon[21325]: VM 724 started with PID 21401 using pve-qemu-kvm_11.0.2-1.
7 16 05:51:14 pve2 pvedaemon[20384]: <root@pam> end task UPID:pve2:0000534D:0000A8F0:6A57F2B6:qmstart:724:root@pam: WARNINGS: 1
 
You must log in or register to reply here.
Top Bottom
Back