[SOLVED] X99 Extreme4 - GPU Passthrough Windows 10 with Radeon RX570 (Black screen)

[sutlnet]

It is not related to vendor-reset. It's the second function of the GPU: the audio device, which used the snd_hda_inte driver. You might want to bind that to vfio-pci (like you did for the audio device of the GT710). It should not be a problem, unless you use the audio of the GPU and experience problems with it.
Did you get vendor-reset working for the AMD GPU (VGA function)? Do you see output on a physical display connected to the RX570 from the Ubuntu VM?

Unfortunately, I am not getting a picture out of the GPU when I start Ubuntu - still black screen.

If I type modprobe vendor-reset I am not seeing anything

If I type the following

cat /sys/bus/pci/devices/0000:02:00.0/reset_method

This is the output device_specific

Can I bind 02:00 or should I only bind 02:00:1?

 

[leesteken]

Unfortunately, I am not getting a picture out of the GPU when I start Ubuntu - still black screen.

Did you set Display to none in the VM configuration? I would expect an installed Ubuntu or a Ubuntu LiveCD/installer to work with a passed through RX570 out of the box.
What is the output of journalctl -b 0 | grep vfio-pci after you started the VM for the first time (after a reboot of the Proxmox host)?

If I type modprobe vendor-reset I am not seeing anything

That's good, It means that vendor-reset is loading/loaded. What is the output of lsmod | grep vendor? Don't bother, the other r

If I type the following


This is the output device_specific

That means that your script ran successfully and activated vendor-reset, which must have been loaded otherwise you would have gotten an error.

Can I bind 02:00 or should I only bind 02:00:1?

Only bind 02:00.1 (get the numeric ID via lspci -nns 02:00.1) (the audio function) to vfio-pci because we want the amdgpu driver to load for 02:00.0 (the VGA function).

I really don't understand why this is so difficult to get to work for your system. It works fine on my system with a RX570 with pve-kernel 6.1.10 and vendor-reset.

 

[sutlnet]

It is working!!!!!!

You are the best!!!!!

I will now try Windows 10 and MacOS

 

[sutlnet]

vfio-pci

I have now bind the GPU Audio in /etc/modprobe.d/vfio.conf
added options vfio-pci ids=1002:aaf0

I can use the graphics cards, but no sound on Ubuntu. Windows 10 is still black screen.

 

[sutlnet]

I tried everything, but I am not able to get the RX570 work with Windows 10 and MacOS. My other card GT710 can be passthrough to both systems without any issues. All Linux distos work without any issues (Ubuntu/Fedora/Arch/Debian). I am definitely missing something in the config files.

Now I have setup Nobara (Linux Gaming Distro) for casual gaming.

I would be very happy about any help here - Thank you

 

[leesteken]

I tried everything, but I am not able to get the RX570 work with Windows 10 and MacOS. My other card GT710 can be passthrough to both systems without any issues. All Linux distos work without any issues (Ubuntu/Fedora/Arch/Debian). I am definitely missing something in the config files.

Now I have setup Nobara (Linux Gaming Distro) for casual gaming.

I would be very happy about any help here - Thank you

I noticed that I need to use pc-q35-6.2 (VM, Hardware, Machine, Version) instead of pc-q35-7.1 for WIndows 10, otherwise the AMD graphics drivers give a code 43 with the RX570.

 

[sutlnet]

I noticed that I need to use pc-q35-6.2 (VM, Hardware, Machine, Version) instead of pc-q35-7.1 for WIndows 10, otherwise the AMD graphics drivers give a code 43 with the RX570.

tried Q-35 6.2 / different HDMI cable / black screen I do not receive any error message and all Linux distress work with the RX570 passthrough without any hiccups.

I am running out of any options - greatly appreciate any further suggestions.
Kernel update and vendor -reset did the trick for host-GPU (RX570) passthrough.

Windows 10 / MacOS is not working

I also tried this approach to passthrough an RX570, but I am having a black screen when directly connected to the HDMI port.

 

[sutlnet]

I have tried a lot, but I am not getting Windows 10 to work.

Grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"

dmesg | grep -e DMAR -e IOMMU
0.012840] ACPI: DMAR 0x000000005C28EE68 0000C4 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.012871] ACPI: Reserving DMAR table memory at [mem 0x5c28ee68-0x5c28ef2b] [ 0.143054] DMAR: IOMMU enabled [ 0.388003] DMAR: Host address width 46 [ 0.388004] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0 [ 0.388010] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap d2008c10ef0466 ecap f0205b [ 0.388013] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1 [ 0.388017] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.388019] DMAR: RMRR base: 0x0000005ce1e000 end: 0x0000005ce2dfff [ 0.388021] DMAR: ATSR flags: 0x0 [ 0.388023] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0 [ 0.388025] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 1 [ 0.388027] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 1 [ 0.388028] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000 [ 0.388030] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. [ 0.388031] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting. [ 0.388697] DMAR-IR: Enabled IRQ remapping in xapic mode [ 0.700262] DMAR: No SATC found [ 0.700265] DMAR: IOMMU feature sc_support inconsistent [ 0.700266] DMAR: IOMMU feature dev_iotlb_support inconsistent [ 0.700267] DMAR: dmar0: Using Queued invalidation [ 0.700271] DMAR: dmar1: Using Queued invalidation [ 0.702931] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 5.803146] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

modprobe vendor-reset is working correctly

Windows 10 vm.config
bios: ovmf boot: order=virtio0;ide2;net0;ide0 cores: 8 cpu: host,hidden=1,flags=+pcid efidisk0: VM-SSD:vm-1100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M hostpci0: 0000:01:00,pcie=1 ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K ide2: Synology-Backup:iso/Win10_22H2_EnglishInternational_x64.iso,media=cdrom,size=5969910K machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=7.1.0,ctime=1677160878 name: Windows10 net0: virtio=3E:67:5C:9F:C1:E7,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsihw: virtio-scsi-single smbios1: uuid=9bf11262-39f2-4820-87c2-ee75e7631224 sockets: 1 vga: none virtio0: VM-SSD:vm-1100-disk-1,cache=writeback,iothread=1,size=64G vmgenid: 0946785a-32ca-4f97-bdda-05de0af8a02e

Ubuntu is working fine with GPU passthrough and sound
bios: ovmf boot: order=scsi0;ide2;net0 cores: 4 efidisk0: VM-SSD:vm-1000-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M hostpci0: 0000:01:00,pcie=1 ide2: Synology-Backup:iso/ubuntu-22.04.1-desktop-amd64.iso,media=cdrom,size=3737140K machine: q35 memory: 8192 meta: creation-qemu=7.1.0,ctime=1677153579 name: Ubuntu net0: virtio=6A:CF:48:91:AE:8D,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: VM-SSD:vm-1000-disk-0,iothread=1,size=32G scsihw: virtio-scsi-single smbios1: uuid=f1d11e66-80ca-4a8e-b9a0-8033881b2517 sockets: 1 usb0: host=046d:c52b usb1: host=0738:1107 vga: none vmgenid: 819f35cb-2310-4297-8996-89ede0b04580

I have spend the last weeks going through every thread which involves AMD GPU passthrough. If anyone could help please

 

[leesteken]

I have tried a lot, but I am not getting Windows 10 to work.

bios: ovmf boot: order=virtio0;ide2;net0;ide0 cores: 8 cpu: host,hidden=1,flags=+pcid efidisk0: VM-SSD:vm-1100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M hostpci0: 0000:01:00,pcie=1 ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K ide2: Synology-Backup:iso/Win10_22H2_EnglishInternational_x64.iso,media=cdrom,size=5969910K machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=7.1.0,ctime=1677160878 name: Windows10 net0: virtio=3E:67:5C:9F:C1:E7,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsihw: virtio-scsi-single smbios1: uuid=9bf11262-39f2-4820-87c2-ee75e7631224 sockets: 1 vga: none virtio0: VM-SSD:vm-1100-disk-1,cache=writeback,iothread=1,size=64G vmgenid: 0946785a-32ca-4f97-bdda-05de0af8a02e

Try without the hidde=1,flags+pcid. I have a RX570 and it works fine with a very similar configuration with Windows 10 22H2 on pc-q35-6.2. Only CPU hot-unplug does not work.

 

[sutlnet]

Try without the hidde=1,flags+pcid. I have a RX570 and it works fine with a very similar configuration with Windows 10 22H2 on pc-q35-6.2. Only CPU hot-unplug does not work.

I tried and it did not work.cThe RX570 is showing up under devices with error43.

Any other suggestions

 

[leesteken]

I tried and it did not work.cThe RX570 is showing up under devices with error43.

Any other suggestions

Double check if vendor-reset is working when starting the VM. Reinstall AMD Adrenalin GPU drivers? Reinstall Windows 10 22H2 (with VirtIO drivers)?

 

[sutlnet]

Double check if vendor-reset is working when starting the VM. Reinstall AMD Adrenalin GPU drivers? Reinstall Windows 10 22H2 (with VirtIO drivers)?

I checked vendor-reset with modprobe vendor-reset and it is working

AMD Adrenalin GPU drivers

Reinstalled and still Error 43

Now reinstalling Windows 10 22H2 with VirtIO drivers

 

[leesteken]

I checked vendor-reset with modprobe vendor-reset and it is working

You also need to check journalctl from around the time that you start the VM and check for reset messages to make sure it is working.

 

[sutlnet]

You also need to check journalctl from around the time that you start the VM and check for reset messages to make sure it is working.

reinstalled windows and still Error 43 after installing the drivers

These are the lines I could identify:

Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x270 Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0 Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370 Feb 27 09:27:49 pve kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping d> Feb 27 09:27:50 pve pvedaemon[1872]: <root@pam> end task UPID:pve:00022982:00485449:63FC6981:qmstart:1300:root@> Feb 27 09:27:53 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 09:27:53 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM

I am not able to find any entries about reset except the USB:

Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0> Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1b@0> Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0> Feb 27 10:15:04 pve pvedaemon[1872]: <root@pam> end task UPID:pve:00024BB0:004CA795:6> Feb 27 10:15:07 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 10:15:07 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 10:15:15 pve kernel: usb 2-13: reset full-speed USB device number 2 using xhci> Feb 27 10:15:16 pve kernel: usb 2-14: reset low-speed USB device number 3 using xhci_>

 

[leesteken]

reinstalled windows and still Error 43 after installing the drivers

These are the lines I could identify:

Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x270 Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0 Feb 27 09:27:49 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370 Feb 27 09:27:49 pve kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping d> Feb 27 09:27:50 pve pvedaemon[1872]: <root@pam> end task UPID:pve:00022982:00485449:63FC6981:qmstart:1300:root@> Feb 27 09:27:53 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 09:27:53 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM

I am not able to find any entries about reset except the USB:

Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0> Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1b@0> Feb 27 10:15:03 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0> Feb 27 10:15:04 pve pvedaemon[1872]: <root@pam> end task UPID:pve:00024BB0:004CA795:6> Feb 27 10:15:07 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 10:15:07 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM Feb 27 10:15:15 pve kernel: usb 2-13: reset full-speed USB device number 2 using xhci> Feb 27 10:15:16 pve kernel: usb 2-14: reset low-speed USB device number 3 using xhci_>

Then vendor-reset is not active. This is the same problem I asked you about in post #19.
Load vendor-reset and activate the device_specific reset_method before starting the VM. Check with journalctl if you see messages with AMD_POLARIS10 when starting the VM.
If you don't see those message then vendor-reset is not working and you cannot progress.

 

[sutlnet]

Then vendor-reset is not active. This is the same problem I asked you about in post #19.
Load vendor-reset and activate the device_specific reset_method before starting the VM. Check with journalctl if you see messages with AMD_POLARIS10 when starting the VM.
If you don't see those message then vendor-reset is not working and you cannot progress.

I am not understanding

vendor-reset is in modules

Can you help me out how do I activate it now. I thought is is active

 

[leesteken]

I am not understanding

You got this working for a Ubuntu VM before right? You need to activate vendor-reset for the RX570 each time you (re)start Proxmox and before starting the VM that uses the RX570.

vendor-reset is in modules

Make sure vendor-reset is loaded and check with lsmod | grep vendor-reset. If there is no output, then you need to load vendor-reset with modprobe vendor-reset. If you get an error then you need to (re)install vendor-reset.

Can you help me out how do I activate it now. I thought is is active

Since kernel version 5.15 (Proxmox 7.2), you need to activate vendor-reset every reboot for each device you need it for (as shown here).
In your case, run this before starting the VM: echo device_specific >"/sys/bus/pci/devices/0000:01:00.0/reset_method".
If you get an error, you must make sure vendor-reset is loaded (see above).

 

[sutlnet]

I am not understanding

vendor-reset is in modules

Can you help me out how do I activate it now. I thought is is active

I am receiving this

[ 4.945918] vendor_reset_hook: installed [ 4.946015] systemd[1]: Started Journal Service. [ 4.948304] VFIO - User Level meta-driver version: 0.3 [ 4.951885] systemd-journald[528]: Received client request to flush runtime journal. [ 4.953951] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem [ 4.957742] systemd-journald[528]: File /var/log/journal/430b783b89e9454d89fe234800e5ed52/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 4.976343] vfio_pci: add [1002:67df[ffffffff:ffffffff]] class 0x000000/00000000 [ 5.000301] vfio_pci: add [1002:aaf0[ffffffff:ffffffff]] class 0x000000/00000000

but no AMD_POLARIS10

 

[leesteken]

I am receiving this

[ 4.945918] vendor_reset_hook: installed [ 4.946015] systemd[1]: Started Journal Service. [ 4.948304] VFIO - User Level meta-driver version: 0.3 [ 4.951885] systemd-journald[528]: Received client request to flush runtime journal. [ 4.953951] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem [ 4.957742] systemd-journald[528]: File /var/log/journal/430b783b89e9454d89fe234800e5ed52/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 4.976343] vfio_pci: add [1002:67df[ffffffff:ffffffff]] class 0x000000/00000000 [ 5.000301] vfio_pci: add [1002:aaf0[ffffffff:ffffffff]] class 0x000000/00000000

but no AMD_POLARIS10

Did you do any of the steps I tried to explain? Let's start over again:
Please reboot your Proxmox host (and don't start the VM automatically).
What is the output of cat /proc/cmdline (on the Proxmox host before starting the VM)?
What is the output of lspci -knns 01:00 (on the Proxmox host before starting the VM)?
What is the output of lsmod | grep vendor-reset (on the Proxmox host before starting the VM)?
What is the output of echo device_specific >"/sys/pci/bus/devices/0000:01:00.0/reset_method" (on the Proxmox host before starting the VM)?
What is the output of journalctl -b 0 | grep reset (on the Proxmox host after starting the VM)?

 

[sutlnet]

Restarted Proxmox server

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.15.85-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init

lspci -knns 01:00

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef) Subsystem: Sapphire Technology Limited Radeon RX 570 Pulse 4GB [1da2:e353] Kernel driver in use: vfio-pci Kernel modules: amdgpu 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0] Subsystem: Sapphire Technology Limited Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1da2:aaf0] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

lsmod | grep vendor-reset

no errors - jumps back to prompt

echo device_specific >"/sys/pci/bus/devices/0000:01:00.0/reset_method"

-bash: /sys/pci/bus/devices/0000:01:00.0/reset_method: No such file or directory

journalctl -b 0 | grep reset

Feb 27 12:58:42 pve kernel: vendor_reset: loading out-of-tree module taints kernel. Feb 27 12:58:42 pve kernel: vendor_reset_hook: installed Feb 27 12:58:42 pve systemd-modules-load[530]: Inserted module 'vendor_reset' Feb 27 13:00:24 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing pre-reset Feb 27 13:00:24 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing reset Feb 27 13:00:24 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing post-reset Feb 27 13:00:24 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: reset result = 0 Feb 27 13:00:26 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing pre-reset Feb 27 13:00:26 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing reset Feb 27 13:00:26 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing post-reset Feb 27 13:00:26 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: reset result = 0 Feb 27 13:00:27 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing pre-reset Feb 27 13:00:27 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing reset Feb 27 13:00:27 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: performing post-reset Feb 27 13:00:27 pve kernel: vfio-pci 0000:01:00.0: AMD_POLARIS10: reset result = 0 Feb 27 13:00:38 pve kernel: usb 2-13: reset full-speed USB device number 2 using xhci_hcd Feb 27 13:00:38 pve kernel: usb 2-14: reset low-speed USB device number 3 using xhci_hcd

I am also getting this in the tasks:

kvm: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. kvm: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. TASK OK