[SOLVED] X99 Extreme4 - GPU Passthrough Windows 10 with Radeon RX570 (Black screen)

[sutlnet]

Hello,

I have tried all the follwing tutorials and after three weeks I am lost and cannot get my RX570 to work within Windows 10:

Tutorials:
https://forum.proxmox.com/threads/solved-single-rx570-gpu-passthrough.118530/
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

I am getting this message when I start the VM and the GPU output is just black.

kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: Failed to mmap 0000:02:00.0 BAR 0. Performance may be slow
TASK OK

I do not know what I should do. Please help:

My Hardware:
Asrock X99 Extreme4
Xeon E5-2670 E3 - 96 DDR4 - Sapphire RX570 4GB
Kernel 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z)
PVE 7.3-4

Grub: I have tried all the different one which I have commented out

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nofb nomodeset video=vesafbff video=efifbff video=simplefbff"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"
GRUB_CMDLINE_LINUX=""


VM.confi: /etc/pve/qemu-server/3010.conf
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=scsi0;net0;ide0
cores: 10
cpu: host,hidden=1,flags=+pcid
efidisk0: Synology-NFS:3010/vm-3010-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:02:00,pcie=1,x-vga=1
ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
machine: pc-q35-6.2
memory: 16484
meta: creation-qemu=7.1.0,ctime=1675867196
name: Windows10Clone
net0: virtio=9A:9B3:3D:55:CB,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: Synology-NFS:3010/vm-3010-disk-1.qcow2,cache=writeback,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=c53d74b0-9ac7-411f-b9fd-a90e9f62e951
sockets: 1
usb0: host=046d:c52b,usb3=1
usb1: host=0738:1107,usb3=1
vmgenid: 29f894bb-dd67-4dd6-96b8-9b594ede6b40

/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1

I am even thinking to buy a new motherboard with GPU or can anyone help me out to solve this.
Thank you so much

 

[leesteken]

If this is the only GPU in the system then you need this work-around. With an AMD GPU it might be worthwhile or switch to a newer kernel version but then you don't bind the GPU to vfio-pci and make sure amdgpu loads for the GPU. And you probably need to install vendor-reset for this generations of AMD GPUs and you need to activate it each reboot.

Please remove all the unnecessary NVidia work-arounds in the configuration, it might even interfere with AMD drivers. If you use a newer kernel, then you don't need to do anything (undo all your changes) but enable IOMMU for Intel and make sure vendor-reset is working. Lots of threads on this forum about single GPU passthrough and AMD GPU vendor-reset.

 

[sutlnet]

If this is the only GPU in the system then you need this work-around. With an AMD GPU it might be worthwhile or switch to a newer kernel version but then you don't bind the GPU to vfio-pci and make sure amdgpu loads for the GPU. And you probably need to install vendor-reset for this generations of AMD GPUs and you need to activate it each reboot.

Please remove all the unnecessary NVidia work-arounds in the configuration, it might even interfere with AMD drivers. If you use a newer kernel, then you don't need to do anything (undo all your changes) but enable IOMMU for Intel and make sure vendor-reset is working. Lots of threads on this forum about single GPU passthrough and AMD GPU vendor-reset.

Thank you for your reply - I do have another GPU (GT710) and passthrough with that card is working fine. Should it still switch to a newer kernel version?

I will now try out your recommendation and report back.

 

[leesteken]

Thank you for your reply - I do have another GPU (GT710) and passthrough with that card is working fine. Should it still switch to a newer kernel version?

My guess is that the AMD is used during boot? I have two AMD GPUs (with one used during boot) and I recommend the newer kernel and vendor-reset (but not blacklisting amdgpu and not binding the GPU to vfio-pci).
If you switch to using the GT710 during boot, you would need the work-around but that makes troubleshooting much harder because you won't see any boot messages

 

[sutlnet]

My guess is that the AMD is used during boot? I have two AMD GPUs (with one used during boot) and I recommend the newer kernel and vendor-reset (but not blacklisting amdgpu and not binding the GPU to vfio-pci).
If you switch to using the GT710 during boot, you would need the work-around but that makes troubleshooting much harder because you won't see any boot messages

I am indeed using the AMD during boot.

 

[leesteken]

I am indeed using the AMD during boot.

I know, because otherwise you would already use the initcall_blacklist=sysfb_init (or the GT710 would not work).

 

[sutlnet]

I know, because otherwise you would already use the initcall_blacklist=sysfb_init (or the GT710 would not work).

So this is what I have to do:

1. Update Kernel (done)
2. non blacklist amdgpu (done)
3. binding the GPU to vfio-pc (currently doing it)
4. vendor-reset

initcall_blacklist=sysfb_init

to be used when I am only using one GPU right.

 

[leesteken]

So this is what I have to do:

1. Update Kernel (done)

Yes

1. non blacklist amdgpu (done)

Yes, make sure it is not blacklisted and amdgpu driver is loaded for the AMD GPU, before starting the VM. Check with lspci -k.

1. binding the GPU to vfio-pc (currently doing it)

No, don't do this for the AMD GPU otherwise amdgpu driver cannot use it. You need the amdgpu driver to free the GPU from boot. Only bind the GT710 as you have done before.

1. vendor-reset

yes, install it and make sure to set the reset_method to device_specific for the AMD GPU after each reboot of the Proxmox host.

initcall_blacklist=sysfb_init

to be used when I am only using one GPU right.

Not exactly. You need it when you want to passthrough the boot GPU unless it is an AMD GPU.

 

[sutlnet]

Yes

Yes, make sure it is not blacklisted and amdgpu driver is loaded for the AMD GPU, before starting the VM. Check with lspci -k.

No, don't do this for the AMD GPU otherwise amdgpu driver cannot use it. You need the amdgpu driver to free the GPU from boot. Only bind the GT710 as you have done before.

yes, install it and make sure to set the reset_method to device_specific for the AMD GPU after each reboot of the Proxmox host.

Not exactly. You need it when you want to passthrough the boot GPU unless it is an AMD GPU

I have done everything and I do not have any error message, but still black screen when Windows VM is booted.

Do I have to change anything in the config file?

 

[leesteken]

I have done everything and I do not have any error message, but still black screen when Windows VM is booted.

Do I have to change anything in the config file?

I have no experience with Windows. Please show the output of cat /proc/cmdline and the VM configuration file (again). What do the files in /etc/modprobe.d/ look like?

 

[sutlnet]

I have no experience with Windows. Please show the output of cat /proc/cmdline and the VM configuration file (again). What do the files in /etc/modprobe.d/ look like?

BOOT_IMAGE=/boot/vmlinuz-6.1.6-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

VM config
#args%3A -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' bios: ovmf boot: order=scsi0;net0;ide0 cores: 10 cpu: host,hidden=1,flags=+pcid efidisk0: Synology-NFS:3010/vm-3010-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:02:00,pcie=1 ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K machine: pc-q35-6.2 memory: 16484 meta: creation-qemu=7.1.0,ctime=1675867196 name: Windows10Clone net0: virtio=9A:9B:D3:3D:55:CB,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi0: Synology-NFS:3010/vm-3010-disk-1.qcow2,cache=writeback,iothread=1,size=64G scsihw: virtio-scsi-single smbios1: uuid=c53d74b0-9ac7-411f-b9fd-a90e9f62e951 sockets: 1 usb0: host=046d:c52b,usb3=1 usb1: host=0738:1107,usb3=1 vmgenid: 29f894bb-dd67-4dd6-96b8-9b594ede6b40

mod probe.d
these are files, but all empty
dkms.conf
kvm.conf
vfio.conf

 

[leesteken]

BOOT_IMAGE=/boot/vmlinuz-6.1.6-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

Looks fine

VM config

#args%3A -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

Indeed, don't use this.

bios: ovmf
boot: order=scsi0;net0;ide0
cores: 10
cpu: host,hidden=1,flags=+pcid

No need for hidden=1

efidisk0: Synology-NFS:3010/vm-3010-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:02:00,pcie=1

Look good, no Primary GPU.

ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
machine: pc-q35-6.2
memory: 16484

Why not use a normal amount like 16384? Note that all VM memory must be pinned into actual RAM because of passthrough.

meta: creation-qemu=7.1.0,ctime=1675867196
name: Windows10Clone
net0: virtio=9A:9B3:3D:55:CB,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: Synology-NFS:3010/vm-3010-disk-1.qcow2,cache=writeback,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=c53d74b0-9ac7-411f-b9fd-a90e9f62e951
sockets: 1
usb0: host=046d:c52b,usb3=1
usb1: host=0738:1107,usb3=1
vmgenid: 29f894bb-dd67-4dd6-96b8-9b594ede6b40

I don't see vga: none, which you want to make the GPU the only one in the VM. This could explain the black screen on the physical display.

mod probe.d
these are files, but all empty
dkms.conf
kvm.conf
vfio.conf

There is no need for empty files. I would expect at least one file for the GT710 with options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1.

Are you sure vendor-reset is installed and activated and working for the AMD GPU? How did you make sure that device_specific is activated for the reset_method after each reboot? Did you add vendor-reset to /etc/modules and run update-initramfs -u?

 

[sutlnet]

vga: none,

I have now updated the file - does this looks fine to you with the via: none

#args%3A -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' bios: ovmf boot: order=scsi0;net0;ide0 cores: 10 cpu: host,flags=+pcid efidisk0: Synology-NFS:3010/vm-3010-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:02:00,pcie=1 ide0: Synology-Backup:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K machine: pc-q35-6.2 memory: 16384 meta: creation-qemu=7.1.0,ctime=1675867196 name: Windows10Clone net0: virtio=9A:9B:D3:3D:55:CB,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi0: Synology-NFS:3010/vm-3010-disk-1.qcow2,cache=writeback,iothread=1,size=64G scsihw: virtio-scsi-single smbios1: uuid=c53d74b0-9ac7-411f-b9fd-a90e9f62e951 sockets: 1 usb0: host=046d:c52b,usb3=1 usb1: host=0738:1107,usb3=1 vmgenid: 29f894bb-dd67-4dd6-96b8-9b594ede6b40 vga: none

 

[leesteken]

I have now updated the file - does this looks fine to you with the via: none

Yes.
Are you sure vendor-reset is installed and activated and working for the AMD GPU? How did you make sure that device_specific is activated for the reset_method after each reboot? Did you add vendor-reset to /etc/modules and run update-initramfs -u?

 

[sutlnet]

Are you sure vendor-reset is installed and activated and working for the AMD GPU? How did you make sure that device_specific is activated for the reset_method after each reboot? Did you add vendor-reset to /etc/modules and run update-initramfs -u?

I have used the links you gave me from instal vendor reset and then I created script

 

[sutlnet]

Yes.
Are you sure vendor-reset is installed and activated and working for the AMD GPU? How did you make sure that device_specific is activated for the reset_method after each reboot? Did you add vendor-reset to /etc/modules and run update-initramfs -u?

root@pve:~# cat /etc/modules # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. vfio vfio_iommu_type1 vfio_pci vfio_virqfd vendor-reset

Does this looks good to you?

 

[sutlnet]

I am still having a black screen when connected directly. With Remote Desktop I installed the drivers, but still error 43. Any ideas?

 

[sutlnet]

I have no experience with Windows. Please show the output of cat /proc/cmdline and the VM configuration file (again). What do the files in /etc/modprobe.d/ look like?

I am now setting up an Ubuntu VM to test if I have the same issue to better troubleshot. Fingers crossed

 

[leesteken]

I am now setting up an Ubuntu VM to test if I have the same issue to better troubleshot. Fingers crossed

That should work. Check in journalctl if you see messages like AMD_POLARIS10: reset result = 0. If you don't see several lines like that when starting the VM, then vendor-reset is not setup correctly.

 

[sutlnet]

leesteken - Thank you so much for all your support not only for me also for everyone in this forum. ​