ISSUE - PVE 8.1.4 - Radeon 6800XT passtrough to Windows 7 guest

T

TheBinaryShadow

New Member
Mar 15, 2024
9
0
1
bshadow.us
Hi everyone,

I spent the last 6 days and night trying to get a GPU passtrough to work.
I'm hoping someone has an ace up their sleve to save me some more agony.

Issue:
Error 43 in device manager

When I add x-vga=1 the VM fails to boot crashing/freezing on Windows boot logo
I tried connecting to the VM while it's in this state over RDP and it was not resposive.

(if I rebild the VM with SeaBIOS and i440fx machine settings then I usually get Error 12)


Setup:
Ryzen 9 5950x
Radeon RX 6800XT

IOMMU - ON
(BIOS and PVE - verified)
ABOVE 4G - ON
(has no effect on outcome)
SR-IOV - ON
Resizable BAR - OFF

PVE 8.1.4 is booted in UEFI mode.

Windows 7 (amd64) is fully updated with all ESU updates as well.

No issues on Windows 11 VM with the same GPU in the same host.

Other PCI passtrough to Windows 7 VMs works flawlesly. (eg. NVMe drive)



/etc/default/grub

Code: 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init noats nobar text"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"


/etc/pve/qemu-server/103.conf

Code: 
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 32
cpu: host
efidisk0: local-lvm:vm-103-disk-0,efitype=4m,size=4M
hostpci0: 0000:0b:00,pcie=1,romfile=vbios.bin
hotplug: 0
machine: pc-q35-8.1
memory: 24576
meta: creation-qemu=8.1.5,ctime=1710181928
name: Dog
net0: virtio=BC:24:11:F1:AA:5B,bridge=vmbr0
numa: 0
ostype: win7
scsi0: Howling:vm-103-disk-0,backup=0,iothread=1,size=250G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=bbc0932d-786d-4793-ba68-7f1acf662eba
sockets: 1
tablet: 0
usb0: host=3-3
usb1: host=1-4
usb2: host=3-4
vmgenid: bbed22b3-378d-4a20-9be4-986386b2d468

cat /etc/modules

Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

vfio
vfio_iommu_type1
vfio_pci

vendor-reset

lspci -nnk

Code: 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Root Complex [1043:8808]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse IOMMU [1043:8808]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
        DeviceName:  Onboard IGD
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1043:8808]
        Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1043:8808]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
        Subsystem: ASUSTeK Computer Inc. FCH SMBus Controller [1043:87c0]
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: ASUSTeK Computer Inc. FCH LPC Bridge [1043:87c0]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
        Subsystem: Samsung Electronics Co Ltd SSD 980 PRO [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
        Kernel driver in use: pcieport
03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge [1043:8808]
        Kernel driver in use: pcieport
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: ASUSTeK Computer Inc. PRIME B450M-A Motherboard [1043:8677]
        Kernel driver in use: r8169
        Kernel modules: r8169
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
        Subsystem: Realtek Semiconductor Co., Ltd. TP-Link TG-3468 v4.0 Gigabit PCI Express Network Adapter [10ec:8168]
        Kernel driver in use: vfio-pci
        Kernel modules: r8169
06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Reserved SPP [1043:8808]
06:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: ASUSTeK Computer Inc. Matisse USB 3.0 Host Controller [1043:8808]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
        Kernel driver in use: ahci
        Kernel modules: ahci
08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
        Kernel driver in use: ahci
        Kernel modules: ahci
09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
        Kernel driver in use: pcieport
0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Kernel driver in use: pcieport
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
        Subsystem: XFX Limited Speedster MERC 319 AMD Radeon RX 6800 XT Black [1eae:6701]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse PCIe Dummy Function [1043:8808]
0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Reserved SPP [1043:8808]
0d:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse Cryptographic Coprocessor PSPCPP [1043:8808]
        Kernel driver in use: ccp
        Kernel modules: ccp
0d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: ASUSTeK Computer Inc. Matisse USB 3.0 Host Controller [1043:87c0]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
0d:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
        Subsystem: ASUSTeK Computer Inc. Starship/Matisse HD Audio Controller [1043:87bb]
        Kernel modules: snd_hda_intel
 
Last edited:
Hi!
Does this card require rom file ?
Code: 
hostpci0: 0000:0b:00,pcie=1,romfile=vbios.bin

Also, this GPU card (Radeon RX 6800XT) has no driver for windows 7.
 
Last edited:
emunt6 said:
Hi!
Does this card require rom file ?
Code: 
hostpci0: 0000:0b:00,pcie=1,romfile=vbios.bin

Also, this GPU card (Radeon RX 6800XT) has no driver for windows 7.
Click to expand...
Hey there, thanks for you reply!

Official AMD site has drivers for Windows 7. Also, I was running Windows 7 on this card directly without ProxMox for testing with no issues.
https://www.amd.com/en/support/grap.../amd-radeon-6800-series/amd-radeon-rx-6800-xt

When it comes to rom file, on Windows 10 and Windows 11 it works without rom file, so I suspect it's not needed. But given now 7 days or troubleshooting I tried that too just to try it. I have no idea what to try next.

I also tried pinning it to a specific PCIe address. Also no change.
 
I would expect a Radeon 6800 and higher to work without any work-arounds. Maybe undo everything except the minimally required PCIe passthrough setup and test with the latest Ubuntu Live installer ISO? Then you'll know that passthrough jsut works and everything else is AMD Windows Driver and Microsoft Windows issues like Machine Type and/or Version.
 
leesteken said:
I would expect a Radeon 6800 and higher to work without any work-arounds. Maybe undo everything except the minimally required PCIe passthrough setup and test with the latest Ubuntu Live installer ISO? Then you'll know that passthrough jsut works and everything else is AMD Windows Driver and Microsoft Windows issues like Machine Type and/or Version.
Click to expand...
Works with no workarounds at all with Windows 10 and Windows 11.

I tried standard setup for Windows 7 as well but no success. This led to a 6 days session of trying everying you can think of.

Drivers work when I install Windows 7 directly on the hardware without PVE.
 
TheBinaryShadow said:
Works with no workarounds at all with Windows 10 and Windows 11.

I tried standard setup for Windows 7 as well but no success. This led to a 6 days session of trying everying you can think of.

Drivers work when I install Windows 7 directly on the hardware without PVE.
Click to expand...
I did not realize, thank you for setting me straight. Maybe reduce the machine version (to a few years back) as they newer PCI(e) layouts can break the expectations of Windows and drivers. Try 4.0 or something instead of 8.1. Maybe even try i440fx instead of q35 for the machine type, which probably requires a reinstall of Windows 7.
 
leesteken said:
I did not realize, thank you for setting me straight. Maybe reduce the machine version (to a few years back) as they newer PCI(e) layouts can break the expectations of Windows and drivers. Try 4.0 or something instead of 8.1. Maybe even try i440fx instead of q35 for the machine type, which probably requires a reinstall of Windows 7.
Click to expand...
I believe I did try that combination before, but I'll give it another go.

At this point I have 3 or 4 VMs. Some with SeaBIOS, some OVMF. I'll report back soon.
 
TheBinaryShadow said:
I believe I did try that combination before, but I'll give it another go.

At this point I have 3 or 4 VMs. Some with SeaBIOS, some OVMF. I'll report back soon.
Click to expand...
You did it! Well... not fully, but you helped a lot! Thank you!

Changing the version to 4.0 but leaving it as a q35 machine fails to boot.
Changing the version to 4.0 and changing the machine type to i440fx also fails to boot.
But... if I remove x-vga=1 argument, it works!

If I add x-vga=1 back, it fails to boot.

This was just a preliminary testing, but it looks solid so far.

I'm sure I tried older versions, not sure if I tried 4.0 specifically, but I did try older versions and x440fx, but I think I was always setting it as primary adapter (x-vga=1) and never got it to work.

Wonder why x-vga=1 would make it fail to boot.

Changing the config to the following works!
Code: 
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 32
cpu: host
efidisk0: local-lvm:vm-103-disk-0,efitype=4m,size=4M
hostpci0: 0000:0b:00
hotplug: 0
machine: pc-i440fx-4.0
memory: 24576
meta: creation-qemu=8.1.5,ctime=1710181928
name: Dog
net0: virtio=BC:24:11:F1:AA:5B,bridge=vmbr0
numa: 0
ostype: win7
scsi0: Howling:vm-103-disk-0,size=250G
scsihw: virtio-scsi-single
smbios1: uuid=bbc0932d-786d-4793-ba68-7f1acf662eba
sockets: 1
tablet: 0
usb0: host=3-3
usb1: host=1-4
usb2: host=3-4
vga: none
vmgenid: bbed22b3-378d-4a20-9be4-986386b2d468
 
Yup! Tested for a couple more hours. Seems rock stable.

Seems to work only with 4.0 and 4.1. Didn't try lower versions, just higher.
OVMF is a must. SeaBIOS just does not work.

Also, if I leave the "Display" on "Default" it still takes over on the GPU as soon as it boots.
After that the Default display KVM display shows as error 43 and doesn't work.

But it works!

Will test more and report. Found quite a few people looking for this online.
Doesn't seem anyone got it to work. Maybe I can help someone else out.

1710643099444.png
 
Last edited:
TheBinaryShadow said:
But... if I remove x-vga=1 argument, it works!

If I add x-vga=1 back, it fails to boot.
Click to expand...
x-vga=1 (Primary GPU) is a work-around for NVidia GPUs and should not be used with Radeon.

TheBinaryShadow said:
Seems to work only with 4.0 and 4.1. Didn't try lower versions, just higher.
Click to expand...
4.0 was just a guess. Maybe try to find the latest that works (or find out which version was current while Windows 7 was supported)?
 
Last edited:
TheBinaryShadow said:
Seems to work only with 4.0 and 4.1. Didn't try lower versions, just higher.
OVMF is a must. SeaBIOS just does not work.

Also, if I leave the "Display" on "Default" it still takes over on the GPU as soon as it boots.
After that the Default display KVM display shows as error 43 and doesn't work.
Click to expand...

- Disable Secure Boot in VMs BIOS if you get error 43.
 
leesteken said:
x-vga=1 (Primary GPU) is a work-around for NVidia GPUs and should not be used with Radeon.


4.0 was just a guess. Maybe try to find the latest that works (or find out which version was current while Windows 7 was supported)?
Click to expand...
Good to know! Thanks!

Yes, I'm going back and forth testing this trying to document everything properly for a full writeup.
5.0 breaks it, anything before is good as far as I can tell so far.

But your shot in the dark, 4.0, seems to really be the best option with no issues at all.

There is some benchmarks and leaving stuff running over a longer period to be sure it's all good, but so far seems rock stable with 4.0.
 
emunt6 said:
- Disable Secure Boot in VMs BIOS if you get error 43.
Click to expand...
Thanks for your reply, Secure Boot was disabled from the get go. It's almost impossible to boot any Windows 7 install with it enabled.

After you apply 10 years of updates it boots fine with Secure Boot enabled, but I leave it off as it makes a lot of weird issues disappear.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back