GPU passthrough Win11 - AMD RX5500XT & Core i9 10900

[siulman]

Hello,

I am sorry as I know there's tons os threads on this and I think I did read them all. I am desperated as I've been trying to get this working for many days now and a very extensive number of hours.
I followed the guide here:
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

I am unable to get the driver installed of the GPU in windows nor see the display through my HDMI monitor.
I am running proxmox 7.2-3.
Windows 11 is properly installed and I see the GPU appearing in my "device manager".

When booting, my monitor shows the "proxmox" display with the message: "BdsDxe: loading Boot0007 "Windows Boot Manager" from HD ..." and not progressing from there.

I really need this to work, please help!


My VM conf:

agent: 1 args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' bios: ovmf boot: order=scsi0;net0;ide0 cores: 4 cpu: host,hidden=1,flags=+pcid efidisk0: local:100/vm-100-disk-0.raw,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:03:00,pcie=1,x-vga=1 ide0: local:iso/virtio-win-0.1.225.iso,media=cdrom,size=519568K machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=6.2.0,ctime=1666862592 name: Windows11 net0: virtio=D6:D4:1B:93:5A:2E,bridge=vmbr0,firewall=1 numa: 1 ostype: win11 scsi0: local:100/vm-100-disk-1.raw,cache=writeback,iothread=1,replicate=0,size=64G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=79ae70da-f348-4c0d-a45e-cf0518743f68 sockets: 1 tpmstate0: local:100/vm-100-disk-2.raw,size=4M,version=v2.0 usb0: host=046d:c534,usb3=1 vga: virtio vmgenid: 6b49778d-7808-45f9-afa2-5d75177b9789

My Grub file:
GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off video=simplefb:off video=vesafb:off" GRUB_CMDLINE_LINUX=""

 

[leesteken]

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

You don't need this, probably not even for NVidia.

hostpci0: 0000:03:00,pcie=1,x-vga=1

,x-vga=1 is useful for NVidia but for AMD you usually don't want it and just set Display to none (vga: none).

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off video=simplefb:off video=vesafb:off"

video=efifb:off video=simplefb:off video=vesafb:off don't really do anything anymore. If you want to passthrough the GPU that used during boot, you want initcall_blacklist=sysfb_init instead (which will prevent you from seeing boot messages and you won't have a host console).
You probably don't need this if you can boot your system with the integrated graphics from your CPU, which is a better choice.

You most likely problem is that the 5500XT does not reset properly and you need vendor-reset to work around this. Here is an old guide. I'm surprised you did not find anything about this in your research. If/when you update your Proxmox to kernel 5.15 or later, you also need to do this before starting the VM.

 

[siulman]

First of all, thank you very much for responding and helping me out.

You don't need this, probably not even for NVidia.

,x-vga=1 is useful for NVidia but for AMD you usually don't want it and just set Display to none (vga: none).

--> Ok removed and also put Graphic card to "none"

video=efifb:off video=simplefb:off video=vesafb:off don't really do anything anymore. If you want to passthrough the GPU that used during boot, you want initcall_blacklist=sysfb_init instead (which will prevent you from seeing boot messages and you won't have a host console).
You probably don't need this if you can boot your system with the integrated graphics from your CPU, which is a better choice.

--> Ok, I removed the "video" parameters and added the "initcall_blacklist=sysfb_init". My BIOS was already configured to use CPU as primary GPU (I have an Asus z490 tuf gaming plus as motherboard)

You most likely problem is that the 5500XT does not reset properly and you need vendor-reset to work around this. Here is an old guide. I'm surprised you did not find anything about this in your research. If/when you update your Proxmox to kernel 5.15 or later, you also need to do this before starting the VM.

I did read a lot about "vendor-reset" but this seemed to be related with not beeing able to reboot or come from suspended state of the VM and I'm not even reaching this step and from a fresh boot of proxmox it's not even working either.

I am still stuck at the exact same screen....and I am not able to install the drivers when using RDP. AMD software is asking me to restart but it doesn't...responding to ping but black screen in RDP. Please have a look to the attachements.

I am posting my code with the changes.

Any help is more than welcome :-(


GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init" GRUB_CMDLINE_LINUX=""



agent: 1 bios: ovmf boot: order=scsi0;net0;ide0 cores: 4 cpu: host,hidden=1,flags=+pcid efidisk0: local:100/vm-100-disk-0.raw,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:03:00,pcie=1 ide0: local:iso/virtio-win-0.1.225.iso,media=cdrom,size=519568K machine: pc-q35-6.2 memory: 8192 meta: creation-qemu=6.2.0,ctime=1666862592 name: Windows11 net0: virtio=D6:D4:1B:93:5A:2E,bridge=vmbr0,firewall=1 numa: 1 ostype: win11 scsi0: local:100/vm-100-disk-1.raw,cache=writeback,iothread=1,replicate=0,size=64G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=79ae70da-f348-4c0d-a45e-cf0518743f68 sockets: 1 tpmstate0: local:100/vm-100-disk-2.raw,size=4M,version=v2.0 usb0: host=046d:c534,usb3=1 vga: none vmgenid: 6b49778d-7808-45f9-afa2-5d75177b9789

 

[leesteken]

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"

You don't need initcall_blacklist=sysfb_init if you are using the integrated graphics during boot. But you might need to early bind the RX5500 to vfio-pci, which might need an additional line softdep amdgpu pre: vfio-pci to make sure vfio-pci is loaded before amdgpu.
Alternatively, update your Proxmox and switch to kernel 5.19 in which the amdgpu driver releases the GPU to vfio-pci more gracefully.

cpu: host,hidden=1,flags=+pcid

I didn't notice this before but there is no reason to hide the fact that your are running a VM.

You could try dumping the ROM-file and adding the romfile parameter to the hostpci line.

Are you sure vendor-reset is working? What does journalctl show during the time you start your VM? (Use the arrow keys on the keyboard to find it, don't just copy the first page.) Is there anything mention of NAVI and reset? Did you enable a device_specific reset?

Can you try booting the VM with an Ubuntu 22.04 live installer ISO? That should boot intro a temporary desktop with the open-source amdgpu driver. Then we at least know that the passthrough is working in principle.

 

[siulman]

You don't need initcall_blacklist=sysfb_init if you are using the integrated graphics during boot. But you might need to early bind the RX5500 to vfio-pci, which might need an additional line softdep amdgpu pre: vfio-pci to make sure vfio-pci is loaded before amdgpu.
Alternatively, update your Proxmox and switch to kernel 5.19 in which the amdgpu driver releases the GPU to vfio-pci more gracefully.

I didn't notice this before but there is no reason to hide the fact that your are running a VM.

You could try dumping the ROM-file and adding the romfile parameter to the hostpci line.

Are you sure vendor-reset is working? What does journalctl show during the time you start your VM? (Use the arrow keys on the keyboard to find it, don't just copy the first page.) Is there anything mention of NAVI and reset? Did you enable a device_specific reset?

Can you try booting the VM with an Ubuntu 22.04 live installer ISO? That should boot intro a temporary desktop with the open-source amdgpu driver. Then we at least know that the passthrough is working in principle.

Where do I need to add this softdep amdgpu pre: vfio-pci line? in /etc/modprobe.d/. ?
Ok, I am removing the "hidden=1" from the vm conf.
I didn't implement vendor-reset this time (already tried a couple of times when re-installing proxmox testing over and over...) as I though it should work from a fresh boot at least.

I'll defenitly test an ubuntu 22.04 VM! that's a good idea. I will install the system, give some time and I'll be back.

 

[leesteken]

Where do I need to add this softdep amdgpu pre: vfio-pci line? in /etc/modprobe.d/. ?

Create a file that ends with .conf in /etc/modprobe.d/. For exmaple nano /etc/modprobe.d/local.conf and add the vfio-pci ids=-line with the correct IDs from lspci -nnks 03:00 for both the VGA and the Audio device. You might also need two softdeps:
softdep amdgpu pre: vfio-pci softdep snd_hda_intel pre: vfio-pci.

Ok, I am removing the "hidden=1" from the vm conf.
I didn't implement vendor-reset this time (already tried a couple of times when re-installing proxmox testing over and over...) as I though it should work from a fresh boot at least.

It depends. It's best to make sure echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method is run after each host reboot and before starting the VM.

I'll defenitly test an ubuntu 22.04 VM! that's a good idea. I will install the system, give some time and I'll be back.

I don't think you need to install it. Just see if you get proper output of the physical display when running the Ubuntu live installer desktop.

 

[siulman]

I installed Ubuntu 20.04 and bypassed the PCI GPU. It that's supposed to work "out of the box" it doesn't...
I have a black screen with cursor blinking (see attached).

When reading you comments I am feeling that I probably lack some linux basic knowledge. I am willing to learn but I'll need a bit more guidance!

Is it possible that something out of this guide broke a default behavior that should be there?
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

Should I start from scratch proxmox install?

I can provide any log you want but would like to know if this is doable? Is vendor-reset applying for ubuntu too?

Any guidance, please, assume I don't really know how to implement, I can try!

 

[leesteken]

I installed Ubuntu 20.04 and bypassed the PCI GPU. It that's supposed to work "out of the box" it doesn't...
I have a black screen with cursor blinking (see attached).

That looks like the display from the integrated graphics showing (a little) of the boot of Proxmox. Do you need to switch it to the other cable connected to the 5500XT, or are you using the 5500XT during Proxmox boot accidentally/automatically because the integrated graphics has no cable connected to it?

When reading you comments I am feeling that I probably lack some linux basic knowledge. I am willing to learn but I'll need a bit more guidance!

Is it possible that something out of this guide broke a default behavior that should be there?
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

Should I start from scratch proxmox install?

I don't think that would help much. Don't stress too much, we just need to get each and every step aligned and working; it won't work (at all) until everything is perfect.

I can provide any log you want but would like to know if this is doable? Is vendor-reset applying for ubuntu too?

vendor-reset is for helping your GPU reset properly (which it can but does not do automatically) for VFIO/passthrough. It needs to be installed on your Proxmox and activated each (re)boot and is independent of your VMs or the operating system inside the VMs.

Any guidance, please, assume I don't really know how to implement, I can try!

There may be a bit of confusion about which display you are looking at, whether vendor-reset is installed/activated/working and whether your changes to /etc/modprobe.d/ are applied (by running update-initramfs -u and update-grub).

After rebooting the Proxmox host and before starting the VM: What are the outputs of cat /proc/cmdline and lspci -nnks 03:00?

 

[siulman]

That looks like the display from the integrated graphics showing (a little) of the boot of Proxmox. Do you need to switch it to the other cable connected to the 5500XT, or are you using the 5500XT during Proxmox boot accidentally/automatically because the integrated graphics has no cable connected to it?

I don't think that would help much. Don't stress too much, we just need to get each and every step aligned and working; it won't work (at all) until everything is perfect.

vendor-reset is for helping your GPU reset properly (which it can but does not do automatically) for VFIO/passthrough. It needs to be installed on your Proxmox and activated each (re)boot and is independent of your VMs or the operating system inside the VMs.

There may be a bit of confusion about which display you are looking at, whether vendor-reset is installed/activated/working and whether your changes to /etc/modprobe.d/ are applied (by running update-initramfs -u and update-grub).

After rebooting the Proxmox host and before starting the VM: What are the outputs of cat /proc/cmdline and lspci -nnks 03:00?

Ok, thanks for the support.

1) I have two monitors connected to the external GPU (5500XT) and no connection at all to the internal GPU. When I start my computer, the BIOS stills shows up on the monitor connected to the external GPU and I see the grub. Do I need a "dummy" HDMI cable connected to the internal GPU?
EDIT: I did it but it's exactly the same as before (dummy cable)

2) This build I am currently running has nothing in terms of vendor-reset. Should I go for the install? The guide I followed last time: https://www.nicksherlock.com/2020/11/working-around-the-amd-gpu-reset-bug-on-proxmox/

3) I didn't change anything in the /etc/modprobe.d/ in my last post, I just did the changes.
--> Done

root@proxmox:/etc/modprobe.d# lspci -n -s 03:00 03:00.0 0300: 1002:7340 (rev c5) 03:00.1 0403: 1002:ab38 root@proxmox:/etc/modprobe.d# cat local.conf options vfio-pci ids=1002:7340,1002:ab38 softdep amdgpu pre: vfio-pci softdep snd_hda_intel pre: vfio-pci

4) What parameters do I need to use here? echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method ?
--> please explain // what is the full command?


5)

root@proxmox:/etc/modprobe.d# update-initramfs -u update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve Running hook script 'zz-proxmox-boot'.. Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace.. No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. root@proxmox:/etc/modprobe.d# update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-5.15.30-2-pve Found initrd image: /boot/initrd.img-5.15.30-2-pve Found memtest86+ image: /boot/memtest86+.bin Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin Adding boot menu entry for EFI firmware configuration done

Is this output OK?

After rebooting the Proxmox host and before starting the VM: What are the outputs of cat /proc/cmdline and lspci -nnks 03:00?

root@proxmox:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt softdep amdgpu pre: vfio-pci root@proxmox:~# lspci -nnks 03:00 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1002:7340] (rev c5) Subsystem: Sapphire Technology Limited Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1da2:e423] Kernel driver in use: vfio-pci Kernel modules: amdgpu 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38] Subsystem: Sapphire Technology Limited Navi 10 HDMI Audio [1da2:e423] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel root@proxmox:~#


--> same status btw

 

[leesteken]

Ok, thanks for the support.

1) I have two monitors connected to the external GPU (5500XT) and no connection at all to the internal GPU. When I start my computer, the BIOS stills shows up on the monitor connected to the external GPU and I see the grub. Do I need a "dummy" HDMI cable connected to the internal GPU?
EDIT: I did it but it's exactly the same as before (dummy cable)

I believe that modern Intel CPUs need an actual display connected to the integrated graphics in order to boot with it. The cable need to be connected to a display but I don't think it needs to be the active/visible display (but it needs to be on or at least stand-by). Your system is using the 5500XT during boot which is causing problems for passthrough.

2) This build I am currently running has nothing in terms of vendor-reset. Should I go for the install? The guide I followed last time: https://www.nicksherlock.com/2020/11/working-around-the-amd-gpu-reset-bug-on-proxmox/

You need to install vendor-reset for a 5500XT, otherwise it will hang when you start and/or reboot the VM.

3) I didn't change anything in the /etc/modprobe.d/ in my last post, I just did the changes.
--> Done

root@proxmox:/etc/modprobe.d# lspci -n -s 03:00 03:00.0 0300: 1002:7340 (rev c5) 03:00.1 0403: 1002:ab38 root@proxmox:/etc/modprobe.d# cat local.conf options vfio-pci ids=1002:7340,1002:ab38 softdep amdgpu pre: vfio-pci softdep snd_hda_intel pre: vfio-pci

This looks correct. (You don't need a softdep in GRUB.)

4) What parameters do I need to use here? echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method ?
--> please explain // what is the full command?

That's the actual command that you run on the Proxmox host console, the Proxmox web GUI shell, your SSH session or a hookscript: echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method (on one line). It changes the reset method of that particular device (03:00.0, your 5500XT) from flr (Function Level Reset) which is bugged to the reset code from vendor-reset.

5)

root@proxmox:/etc/modprobe.d# update-initramfs -u update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve Running hook script 'zz-proxmox-boot'.. Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace.. No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. root@proxmox:/etc/modprobe.d# update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-5.15.30-2-pve Found initrd image: /boot/initrd.img-5.15.30-2-pve Found memtest86+ image: /boot/memtest86+.bin Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin Adding boot menu entry for EFI firmware configuration done

Is this output OK?

Yes, but it also shows that you have not updated your Proxmox (with the no-subscription repository) which I do recommend in general.

root@proxmox:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt softdep amdgpu pre: vfio-pci

You don't need the softdep amdgpu pre: vfio-pci here, it should be in local.conf, which it also is.

root@proxmox:~# lspci -nnks 03:00 [03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1002:7340] (rev c5) Subsystem: Sapphire Technology Limited Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1da2:e423] Kernel driver in use: vfio-pci Kernel modules: amdgpu 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38] Subsystem: Sapphire Technology Limited Navi 10 HDMI Audio [1da2:e423] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel root@proxmox:~#

This looks good (if this is before starting the VM).

The problem seems to be that you are not using vendor-reset and that the integrated graphics is not used for boot (but the passthrough GPU instead).

 

[siulman]

I believe that modern Intel CPUs need an actual display connected to the integrated graphics in order to boot with it. The cable need to be connected to a display but I don't think it needs to be the active/visible display (but it needs to be on or at least stand-by). Your system is using the 5500XT during boot which is causing problems for passthrough.

Ok, I plugged one of the monitors to the iGPU and was able to run windows VM with passthrough on the other monitor... I need to figure out now how to keep connected one monitor to the iGPU without using it. I would say that one of the plugs (the display port maybe) might do the trick?

EDIT: using another entry of the monitor does work, now I can use both monitors as the iGPU has "something plugged"

You need to install vendor-reset for a 5500XT, otherwise it will hang when you start and/or reboot the VM.

Done but not sure it works:

root@proxmox:~/vendor-reset# dkms install . Creating symlink /var/lib/dkms/vendor-reset/0.1.1/source -> /usr/src/vendor-reset-0.1.1 DKMS: add completed. Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area... make -j20 KERNELRELEASE=5.15.30-2-pve KDIR=/lib/modules/5.15.30-2-pve/build... cleaning build area... DKMS: build completed. vendor-reset.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/5.15.30-2-pve/updates/dkms/ depmod... DKMS: install completed.

This looks correct. (You don't need a softdep in GRUB.)

That's the actual command that you run on the Proxmox host console, the Proxmox web GUI shell, your SSH session or a hookscript: echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method (on one line). It changes the reset method of that particular device (03:00.0, your 5500XT) from flr (Function Level Reset) which is bugged to the reset code from vendor-reset.

This command does not work for me:

echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method

Yes, but it also shows that you have not updated your Proxmox (with the no-subscription repository) which I do recommend in general.

I will once I get my setup working as I want :-(

You don't need the softdep amdgpu pre: vfio-pci here, it should be in local.conf, which it also is.

oh ok, I'll change it.

This looks good (if this is before starting the VM).

Yes it is

The problem seems to be that you are not using vendor-reset and that the integrated graphics is not used for boot (but the passthrough GPU instead).

I just installed it but not sure it does the trick.
EDIT: Don't think it works as I powered off/powered on my VM and I get a black screen. Maybe the reason why I can't install the drviers too?



-->> Now when I try to install the drivers AMD in the windows VM the full system crashes (including PVE) and I lose even ping.... not sure what happens. That's funny, the same happened to me when I tested this in ESXI!

Suggestions?

 

[leesteken]

Ok, I plugged one of the monitors to the iGPU and was able to run windows VM with passthrough on the other monitor... I need to figure out now how to keep connected one monitor to the iGPU without using it. I would say that one of the plugs (the display port maybe) might do the trick?

EDIT: using another entry of the monitor does work, now I can use both monitors as the iGPU has "something plugged"

Good! So in principle passthrough is working.

Done but not sure it works:

root@proxmox:~/vendor-reset# dkms install . Creating symlink /var/lib/dkms/vendor-reset/0.1.1/source -> /usr/src/vendor-reset-0.1.1 DKMS: add completed. Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area... make -j20 KERNELRELEASE=5.15.30-2-pve KDIR=/lib/modules/5.15.30-2-pve/build... cleaning build area... DKMS: build completed. vendor-reset.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/5.15.30-2-pve/updates/dkms/ depmod... DKMS: install completed.

Look alright.

This command does not work for me:

echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method

What do you mean "does not work for me"?

I will once I get my setup working as I want :-(


oh ok, I'll change it.


Yes it is



I just installed it but not sure it does the trick.
EDIT: Don't think it works as I powered off/powered on my VM and I get a black screen. Maybe the reason why I can't install the drviers too?


-->> Now when I try to install the drivers AMD in the windows VM the full system crashes (including PVE) and I lose even ping.... not sure what happens. That's funny, the same happened to me when I tested this in ESXI!

Suggestions?

It looks like it's a Windows driver issue. I have no experience with that and cannot help you further.
Does the same VM work when you boot it with the Ubuntu installer? Not to install Ubuntu but to check if you see output on the display.

 

[siulman]

Good! So in principle passthrough is working.

Look alright.

Yes but power off / power on does not work...

What do you mean "does not work for me"?

root@proxmox:~# echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method -bash: echo: write error: Invalid argument

It looks like it's a Windows driver issue. I have no experience with that and cannot help you further.
Does the same VM work when you boot it with the Ubuntu installer? Not to install Ubuntu but to check if you see output on the display.

Sure but how can it crash the whole system!? is it a driver issue or some iommu stuff?

EDIT:

by the way, this is what I see in my journal...thousands of same lines:

Oct 27 17:01:12 proxmox kernel: vfio-pci 0000:03:00.0: BAR 0: can't reserve [mem 0x6000000000-0x600ff>

 

[leesteken]

Yes but power off / power on does not work...

Do you mean shutting down the VM and restarting it? Sounds like vendor-reset is not loaded.

root@proxmox:~# echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method -bash: echo: write error: Invalid argument

Looks like vendor-reset is not loaded. Does running modprobe vendor-reset first resolve this issues? Did you add vendor-reset to /etc/modules?

Sure but how can it crash the whole system!? is it a driver issue or some iommu stuff?

It should not, but in practice it can via the physical PCIe bus.

EDIT:

by the way, this is what I see in my journal...thousands of same lines:

Oct 27 17:01:12 proxmox kernel: vfio-pci 0000:03:00.0: BAR 0: can't reserve [mem 0x6000000000-0x600ff>

Looks like vendor-reset is not loaded (it did appear to install fine). Please see my previous remarks in this post.

 

[siulman]

Do you mean shutting down the VM and restarting it? Sounds like vendor-reset is not loaded.

Looks like vendor-reset is not loaded. Does running modprobe vendor-reset first resolve this issues? Did you add vendor-reset to /etc/modules?

It should not, but in practice it can via the physical PCIe bus.

Looks like vendor-reset is not loaded (it did appear to install fine). Please see my previous remarks in this post.

Yes, I loaded it in the modules:

root@proxmox:/etc/modules-load.d# cat modules.conf # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. vfio vfio_iommu_type1 vfio_pci vfio_virqfd vendor-reset


root@proxmox:/etc/modules-load.d# modprobe vendor-reset modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/5.15.64-1-pve

root@proxmox:/etc/modules-load.d# dkms status vendor-reset, 0.1.1, 5.15.30-2-pve, x86_64: installed


It's probably this... what am I missing?

 

[leesteken]

root@proxmox:/etc/modules-load.d# modprobe vendor-reset modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/5.15.64-1-pve

root@proxmox:/etc/modules-load.d# dkms status vendor-reset, 0.1.1, 5.15.30-2-pve, x86_64: installed

It's probably this... what am I missing?

You build and installed vendor-reset for kernel 5.1.5-30-2 but you are running kernel 5.15.-64-1 (I guess you updated Proxmox and restarted). You need to install vendor-reset for this kernel as well. Unsually, kernel-reset is build automatically when installing a newer kernel but maybe you don't have the right kernel-headers installed?

 

[siulman]

You build and installed vendor-reset for kernel 5.1.5-30-2 but you are running kernel 5.15.-64-1 (I guess you updated Proxmox and restarted). You need to install vendor-reset for this kernel as well. Unsually, kernel-reset is build automatically when installing a newer kernel but maybe you don't have the right kernel-headers installed?

You are absolutelly right.
I uninstall it and re-install it again and now your command: echo device_specific >/sys/bus/pci/devices/0000:03:00.0/reset_method does work as well as the mdprobe you asked: modprobe vendor-reset

I was able to install the AMD driver now! I saw the resolution settings changing as in baremetal environnement. However, when windows asked me to restart to apply changes it got stucked...
I saw an error on the temporary monitor connect to the IGPU, let me share it with you (attached).
I tried to stop the VM from the GUI and it crashed again my PVE!!!

What do you think?

After that...I tried to shutdown my Win11 VM and Start again and it didn't work...black screen, so the vendor-reset is not working? What am I missing? isn't vendor-reset for this? so I can shutdown/start my VM again?

 

[leesteken]

However, when windows asked me to restart to apply changes it got stucked...
I saw an error on the temporary monitor connect to the IGPU, let me share it with you (attached).
I tried to stop the VM from the GUI and it crashed again my PVE!!!

What do you think?

You are getting PCIe errors, which can indeed crash your system. Maybe it's the GPU, maybe it's in combination with your motherboard/CPU. Try upgrading everything to the latest BIOS/firmware versions. I don't know how to troubleshoot such an issue (and Windows complicates matters for me). Maybe search the internet and/or make a new thread for this specific PCIe error issue.

After that...I tried to shutdown my Win11 VM and Start again and it didn't work...black screen, so the vendor-reset is not working? What am I missing? isn't vendor-reset for this? so I can shutdown/start my VM again?

Yes it should, but you are getting PCIe hardware errors.

 

[siulman]

You are getting PCIe errors, which can indeed crash your system. Maybe it's the GPU, maybe it's in combination with your motherboard/CPU. Try upgrading everything to the latest BIOS/firmware versions. I don't know how to troubleshoot such an issue (and Windows complicates matters for me). Maybe search the internet and/or make a new thread for this specific PCIe error issue.

Yes it should, but you are getting PCIe hardware errors.

Thanks a lot for your help then!
You were very patient with me.

 

[leesteken]

Maybe you can check if your Ubuntu VM (which I think you installed before) works better?