[SOLVED] X99 Extreme4 - GPU Passthrough Windows 10 with Radeon RX570 (Black screen)

sutlnet · Feb 10, 2023

leesteken said:
That should work. Check in journalctl if you see messages like AMD_POLARIS10: reset result = 0. If you don't see several lines like that when starting the VM, then vendor-reset is not setup correctly.

Passing through my RX570 is not working with Ubuntu. I will now setup again vendor-reset.

this is my UBUNTU setup:


bios: ovmf
boot: order=scsi0;ide2;net0
cores: 6
efidisk0: VMssd:vm-1000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: Synology-Backup:iso/ubuntu-22.04.1-desktop-amd64.iso,media=cdrom,size=3737140K
machine: q35
memory: 8192
meta: creation-qemu=7.1.0,ctime=1676025834
name: Ubuntu
net0: virtio=F6:2A:4E:D2:D5:B3,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: VMssd:vm-1000-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=712aec89-d82e-4b32-b17c-4ecdbfc0d4c2
sockets: 1
vmgenid: f9e13872-917f-4c82-a0c6-b6bf3dc3e5a3

sutlnet · Feb 10, 2023

sutlnet said:
Passing through my RX570 is not working with Ubuntu. I will now setup again vendor-reset.

this is my UBUNTU setup:

bios: ovmf boot: order=scsi0;ide2;net0 cores: 6 efidisk0: VMssd:vm-1000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M ide2: Synology-Backup:iso/ubuntu-22.04.1-desktop-amd64.iso,media=cdrom,size=3737140K machine: q35 memory: 8192 meta: creation-qemu=7.1.0,ctime=1676025834 name: Ubuntu net0: virtio=F6:2A:4E:D2:D5:B3,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: VMssd:vm-1000-disk-1,iothread=1,size=32G scsihw: virtio-scsi-single smbios1: uuid=712aec89-d82e-4b32-b17c-4ecdbfc0d4c2 sockets: 1 vmgenid: f9e13872-917f-4c82-a0c6-b6bf3dc3e5a3

GPU is not working as well with UBUNTU. I did the vendor-reset on Proxmox and I created this script:

lib/systemd/system/vrwa.service:
[Unit]
Description=vrwa Service
After=multi-user.target
[Service]
ExecStart=/usr/bin/bash -c 'echo device_specific > /sys/bus/pci/devices/0000:0c:00.0/reset_method'
[Install]
WantedBy=multi-user.target
and i enabled it with systemd: systemctl enable vrwa

I am probably missing something. How do I run journalctl as the output gives me strange dates

leesteken · Feb 10, 2023

sutlnet said:
I am probably missing something. How do I run journalctl as the output gives me strange dates

Run journalctl -e to start at the end and use arrow keys and PageUp and PageDown to scroll to the right day and time.
Use journalctl -b 0 | grep vfio-pci to see all vfio-pci messages since the last reboot. It should contain reset messages from vendor-reset.

What is the output of lspci -nnks 0c:00 after a Proxmox reboot and before starting the VM?
What is your VM configuration with passthrough (the last one you showed did not have a hostpci0 line)?

sutlnet · Feb 10, 2023

leesteken said:
Run journalctl -e to start at the end and use arrow keys and PageUp and PageDown to scroll to the right day and time.
Use journalctl -b 0 | grep vfio-pci to see all vfio-pci messages since the last reboot. It should contain reset messages from vendor-reset.

What is the output of lspci -nnks 0c:00 after a Proxmox reboot and before starting the VM?
What is your VM configuration with passthrough (the last one you showed did not have a hostpci0 line)?

Is this what you mean:

Feb 10 14:10:04 pve pvescheduler[1959]: starting server
Feb 10 14:10:04 pve systemd[1]: Started Proxmox VE scheduler.
Feb 10 14:10:04 pve systemd[1]: Reached target Multi-User System.
Feb 10 14:10:04 pve systemd[1]: Reached target Graphical Interface.
Feb 10 14:10:04 pve systemd[1]: Starting Update UTMP about System Runlevel Changes...
Feb 10 14:10:04 pve systemd[1]: Started vrwa Service.
Feb 10 14:10:04 pve bash[1963]: /usr/bin/bash: line 1: echo: write error: Invalid argument
Feb 10 14:10:04 pve systemd[1]: vrwa.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 14:10:04 pve systemd[1]: vrwa.service: Failed with result 'exit-code'.
Feb 10 14:10:04 pve kernel: amdgpu 0000:02:00.0: Unsupported reset method 'device_specific'
Feb 10 14:10:04 pve systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Feb 10 14:10:04 pve systemd[1]: Finished Update UTMP about System Runlevel Changes.
Feb 10 14:10:04 pve systemd[1]: Startup finished in 4.335s (kernel) + 10.136s (userspace) = 14.472s.
Feb 10 14:10:06 pve chronyd[1694]: Selected source 212.18.3.19 (2.debian.pool.ntp.org)
Feb 10 14:10:06 pve chronyd[1694]: System clock TAI offset set to 37 seconds
Feb 10 14:10:10 pve kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://>
Feb 10 14:10:10 pve kernel: FS-Cache: Loaded

For this journalctl -b 0 | grep vfio-pci I am not getting anything out

Ubuntu Config

Code:

  GNU nano 5.4                             /etc/pve/qemu-server/1000.conf                                       
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 6
efidisk0: VMssd:vm-1000-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:02:00,pcie=1
ide2: Synology-Backup:iso/ubuntu-22.04.1-desktop-amd64.iso,media=cdrom,size=3737140K
machine: q35
memory: 8192
meta: creation-qemu=7.1.0,ctime=1676025834
name: Ubuntu
net0: virtio=F6:2A:4E:D2:D5:B3,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: VMssd:vm-1000-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=712aec89-d82e-4b32-b17c-4ecdbfc0d4c2
sockets: 1
vmgenid: f9e13872-917f-4c82-a0c6-b6bf3dc3e5a3

leesteken · Feb 10, 2023

sutlnet said:
Feb 10 14:10:04 pve systemd[1]: Started vrwa Service.
Feb 10 14:10:04 pve bash[1963]: /usr/bin/bash: line 1: echo: write error: Invalid argument
Feb 10 14:10:04 pve systemd[1]: vrwa.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 14:10:04 pve systemd[1]: vrwa.service: Failed with result 'exit-code'.
Feb 10 14:10:04 pve kernel: amdgpu 0000:02:00.0: Unsupported reset method 'device_specific'

Looks like vendor-reset is not loaded. What is the output of lsmod | grep vendor? Is vendor-reset in /etc/modules? Did you run update-initramfs -u? Does your script run before kernel modules are loaded maybe? Try running the command manually after running modprobe vendor-reset to load vendor-reset.

sutlnet said:
For this journalctl -b 0 | grep vfio-pci I am not getting anything out

That's probably because vendor-reset is not loaded and you did not start a VM with passthrough.
What is the output of lspci -nnks 02:00 after a Proxmox reboot and before starting the VM?
Then start the VM and run journalctl -b 0 | grep vfio-pci afterwards.

sutlnet · Feb 10, 2023

leesteken said:
/etc/modules


root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vendor-reset

leesteken said:
lsmod | grep vendor

I do not getting any output

leesteken said:
update-initramfs -u

I did run it

leesteken said:
lspci -nnks 02:00


lspci -nnks 02:00
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
        Subsystem: Sapphire Technology Limited Radeon RX 570 Pulse 4GB [1da2:e353]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
        Subsystem: Sapphire Technology Limited Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1da2:aaf0]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

leesteken said:
journalctl -b 0 | grep vfio-pci


root@pve:~# journalctl -b 0 | grep vfio-pci
Feb 10 16:04:15 pve kernel: vfio-pci 0000:02:00.0: vgaarb: deactivate vga console
Feb 10 16:04:15 pve kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370

leesteken · Feb 10, 2023

sutlnet said:
lspci -nnks 02:00 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef) Subsystem: Sapphire Technology Limited Radeon RX 570 Pulse 4GB [1da2:e353] Kernel driver in use: amdgpu Kernel modules: amdgpu 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0] Subsystem: Sapphire Technology Limited Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1da2:aaf0] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel

This looks good.

sutlnet said:
root@pve:~# journalctl -b 0 | grep vfio-pci Feb 10 16:04:15 pve kernel: vfio-pci 0000:02:00.0: vgaarb: deactivate vga console Feb 10 16:04:15 pve kernel: vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x19@0x270 Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0 Feb 10 16:04:17 pve kernel: vfio-pci 0000:02:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370

This also shows that vendor-reset is not working.

sutlnet said:
I do not getting any output

I did run it

Vendor-reset is not loaded and therefore the AMD GPU will not reset properly for passthrough.

What is the output of modprobe vendor-reset?
What is the output of cat /sys/bus/pci/devices/0000:02:00.0/reset_method?

sutlnet · Feb 10, 2023

leesteken said:
modprobe vendor-rese

I am getting an error:


root@pve:~# modprobe vendor-rese
modprobe: FATAL: Module vendor-rese not found in directory /lib/modules/6.1.6-1-pve

leesteken said:
cat /sys/bus/pci/devices/0000:02:00.0/reset_method


root@pve:~# cat /sys/bus/pci/devices/0000:02:00.0/reset_method
bus

I did all the following steps from NickSherlocks' page:


# Get latest Proxmox kernel headers:
apt install pve-headers

# Did that fail? If so make sure you have Proxmox repository set up properly! https://pve.proxmox.com/wiki/Package_Repositories

# Get required build tools:
apt install git dkms build-essential

# Perform the build:
git clone https://github.com/gnif/vendor-reset.git
cd vendor-reset
dkms install .

# Enable vendor-reset to be loaded automatically on startup:
echo "vendor-reset" >> /etc/modules
update-initramfs -u

# Reboot to load the module:
shutdown -r now

Struggling with the getup script

leesteken · Feb 10, 2023

sutlnet said:
I am getting an error:

root@pve:~# modprobe vendor-rese modprobe: FATAL: Module vendor-rese not found in directory /lib/modules/6.1.6-1-pve

It's vendor-reset, not vendor-rese. Sorry, I must have typed it wrong. What is the output of modprobe vendor-reset?
What is the output of cat /sys/bus/pci/devices/0000:02:00.0/reset_method afterwards?

sutlnet · Feb 10, 2023

leesteken said:
modprobe vendor-reset

I am still getting this error message:

modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/6.1.6-1-pve

leesteken said:
cat /sys/bus/pci/devices/0000:02:00.0/reset_method

It just says bus


root@pve:~# cat /sys/bus/pci/devices/0000:02:00.0/reset_method
bus

leesteken · Feb 10, 2023

sutlnet said:
I am still getting this error message:

modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/6.1.6-1-pve

Looks like you did not build the module for this kernel. Maybe you updated to 6.1 after installing vendor-reset? Did you also install pve-headers-6.1? Otherwise it cannot build the vendor-reset module for pve-kernel-6.1.6-1-pve.

sutlnet · Feb 10, 2023

leesteken said:
Looks like you did not build the module for this kernel. Maybe you updated to 6.1 after installing vendor-reset? Did you also install pve-headers-6.1? Otherwise it cannot build the vendor-reset module for pve-kernel-6.1.6-1-pve.

I just ran everyone and still getting the fatal error

sutlnet · Feb 10, 2023

sutlnet said:
I just ran everyone and still getting the fatal error

I have vendor reset, but not under


/lib/modules/6.1.6-1-pve

it is under


/vendor/reset

Can I just copy the directory into


/lib/modules/6.1.6-1-pve

leesteken · Feb 10, 2023

sutlnet said:
I have vendor reset, but not under
/lib/modules/6.1.6-1-pve
it is under
/vendor/reset

That is probably the source code, not the compiled module.

sutlnet said:
Can I just copy the directory into
/lib/modules/6.1.6-1-pve

Did that work?
Did you install the right headers with apt install pve-headers-6.1.6-1-pve?
What is the output of dkms install . when you run it from the directory where you cloned the vendor-reset git repository?

sutlnet · Feb 10, 2023

leesteken said:
apt install pve-headers-6.1.6-1-pve

I have now installed it and run update-initramfs -u

leesteken said:
dkms install .

this is the output


Error! DKMS tree already contains: vendor-reset-0.1.1

should I now reboot and try with ubuntu

leesteken · Feb 10, 2023

sutlnet said:
I have now installed it and run update-initramfs -u

this is the output

Error! DKMS tree already contains: vendor-reset-0.1.1

should I now reboot and try with ubuntu

Maybe it will work. Or maybe you need to remove the module first. I'm not sure. Does modprobe vendor-reset now work?

If you run apt update and apt install pve-headers-6.1 and apt dist-upgrade, you'll get a new kernel version 6.1.10 (from today) and vendor-reset should be installed correctly the first time.

sutlnet · Feb 10, 2023

leesteken said:
modprobe vendor-reset

Still getting this message
modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/6.1.6-1-pve

I will now remove the module and upgrade - and then install vendor-reset again

Thank you - I will give an update

leesteken · Feb 10, 2023

sutlnet said:
Still getting this message
modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/6.1.6-1-pve

I will now remove the module and upgrade - and then install vendor-reset again

Thank you - I will give an update

Don't remove it! Just do the upgrade steps: apt update, apt install pve-headers-6.1 and apt dist-upgrade. You can remove all the 6.1.6 stuff later.

sutlnet · Feb 10, 2023

leesteken said:
Don't remove it! Just do the upgrade steps: apt update, apt install pve-headers-6.1 and apt dist-upgrade. You can remove all the 6.1.6 stuff later.

Updated and started Ubuntu VM, but I am getting this message


kvm: vfio: Cannot reset device 0000:02:00.1, no available reset mechanism.
kvm: vfio: Cannot reset device 0000:02:00.1, no available reset mechanism.
TASK OK

I assume that vendor reset is not active. I am struggling with vendor reset. I followed all the steps from Nick Sherlock's page. Do you have any idea?

leesteken · Feb 10, 2023

sutlnet said:
Updated and started Ubuntu VM, but I am getting this message

kvm: vfio: Cannot reset device 0000:02:00.1, no available reset mechanism. kvm: vfio: Cannot reset device 0000:02:00.1, no available reset mechanism. TASK OK

I assume that vendor reset is not active. I am struggling with vendor reset. I followed all the steps from Nick Sherlock's page. Do you have any idea?

It is not related to vendor-reset. It's the second function of the GPU: the audio device, which used the snd_hda_inte driver. You might want to bind that to vfio-pci (like you did for the audio device of the GT710). It should not be a problem, unless you use the audio of the GPU and experience problems with it.
Did you get vendor-reset working for the AMD GPU (VGA function)? Do you see output on a physical display connected to the RX570 from the Ubuntu VM?

[SOLVED] X99 Extreme4 - GPU Passthrough Windows 10 with Radeon RX570 (Black screen)

Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

We value your privacy