Passthrough:

[kobold81]

Funny thing, I got passthrough working on pve 9 with win11 by providing a rom file and a file for audio as well as the additional 2 pci devices (number 2 + 3). For Ubuntu and batocera, hostpci 0 + 1 was enough, only windows needed also 2 +3:

hostpci0: 0000:c7:00.0,pcie=1,romfile=vbios_7xxx.bin,x-vga=1
hostpci1: 0000:c7:00.1,pcie=1,romfile=AMDGopDriver.rom
hostpci2: 0000:c7:00.5,pcie=1
hostpci3: 0000:c7:00.6,pcie=1

I added a second nvme to have a raid 1 and did a reinstall of proxmox and imported the vm backup and recreated the config file changes.
But now, when I start the vm, I get the warning that hostcpi 2 +3 also need to access to more iommu groups. I added these but I fail to have
vfio replace xhci_hcd and xhci_pci:

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:1900,1002:1640,1022:15c7,1022:15b9,1022:15ba,1022:15ba,1022:15e2,1022:15e3
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
softdep xhci_hcd pre: vfio-pci
softdep xhci_pci pre: vfio-pci
softdep ccp pre: vfio

I fail to replace the drivers for the usb controller c7:00.3 & c7:00.4:
c7:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
c7:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ba]
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
Kernel driver in use: xhci_hcd

I do not understand what changed so that I have to add these hostcpi devices and how I have the machne load & use the vfio-pci driver befpore the xhci_* drivers

 

[DerekG]

All the changes made on the host to get the passthrough to work, were lost when you re-installed Proxmox. That includes the rom files which were stored on the host and therefore not backed up as a part o the VM.

You need to follow all the steps which you carried out previously, apart from the changes made to the VM config.

 

[kobold81]

Vbios and sounddriver img are in the kvm subdirectory like on the old machine. The Ubuntu and batocera work with the same settings perfectly, only windows has the need for adding all soundchip devices. On the old system, adding the pci devices was all it took, now these request additional iommu group devices. Xhci needs to be preloaded by vfio, I will try to blacklist Xhci so that vfi gets loaded first.

 

[kobold81]

I found out why it did not work: I imported my 650GB windows volume from the previous proxmox server and it seems it is unreadable.

 

[DerekG]

I found out why it did not work: I imported my 650GB windows volume from the previous proxmox server and it seems it is unreadable.

Glad that you got it working, but why keep a 650GB Windows volume, that must make backing up the VM a nightmare.

 

[kobold81]

@DerekG

That is my Windows Gaming VM

While I can access the drive now (howto) I still do not have igpu passthrough because the machine still demands more iommu groups. But when I add them, the server crashes and restarts

 

[DerekG]

@DerekG

That is my Windows Gaming VM

While I can access the drive now (howto) I still do not have igpu passthrough because the machine still demands more iommu groups. But when I add them, the server crashes and restarts

Put that excess data on another drive, either external (or more difficult an iSCSI drive) which will then appear as 'Drive D:' in Windows, that will help to keep the Windows VM as clean as possible. Hell in my case even 'My Documents' and 'Download' are separate from the VM.

I've been through the AMD iGPU passthrough setup and it works great if you follow the steps exactly. I used the following:

https://github.com/isc30/ryzen-gpu-passthrough-proxmox?tab=readme-ov-file

Once you have that working, then you need to deal with the AMD Reset bug which must be addressed both on the Host and within the guest VM.

 

[kobold81]

I known the repo, but Passthrough works currently perfectly with ubuntu and batocera but on Windows I had to pass c7:00.5 and c7:00.6 before reinstalling the Server. Now, they Need more iommu groups. I will try some alternative ideas, perhaps one sticks

 

[kobold81]

ok, found the solution:

Somehow, the reimported image did not really work. While it booted without passthrough, some programms could not start.
Also, my formerly working vm config does not work for the newest win 11 release.

So, I created a clean, new windows 11 vm.
I tried adding hostpci2: 0000:c7:00.5,pcie=1 and hostpci3: 0000:c7:00.6,pcie=1 but no chance, more iommu groups were asked for. Then I had windows install all available updates.
Then I just added these two pci devicces 7:00.0 and c7:00.1 with the rom and bin as shown in my first post.
And oh wonder, now pasthrough works with the same config as ubuntu and batocera.