Passthrough on 6900xt and iGPU 7950x3d

Malombrado · Oct 15, 2023

Hello everyone,

i'm just using the last Promox version 8, and windows 11 22h2 sept 22

followed the Reddit tutorial to passtrought for my AMD GPU saphire 6900xt, the windows see the gpu but was unbale to make it work (43), it still missing something and i don't know how to resolve this problem.

Tried for 2 entire days on forum posts and reddit asks trying to find a solution but still with no luck since many of them was from previous versions of promox or differents setups.

i'll put my codes here and maybe this help to find the error:
Grub

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

100.conf

Code:

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=0ff"'
bios: ovmf
boot: order=ide2;scsio;neto
cores: 4
cpu: host,hidden=1, flags=+pcid
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpcio: 0000:03:00,pcie=1,x-vga=1
ide2: local:iso/virtio-win-O.1.240.iso,media=cdrom, size=612812K
machine: pc-g35-8.0
memory: 16384
meta: creation-gemu=8.0.2,ctime=1697402213
name: NEW11
neto: virtio=96:B3:87:9F:AB:A6, bridge=wmbr0O, firewall=1
numa: 0
ostype: win11
Sscsi0: local-lvm:vm-100-disk-1,iothread=1,size=120G
Scsihw: virtio-scsi-single
smbios1: uuid=b7c80e53-efe6-4717-94ba-295235aca870
sockets: 1
tpmstate0: local-lvm:vm-100-disk-2,5size=4M,version=v2.0
vga: none
vmgenid: 176683bc-374b-4959-b38f-9cedfb832762

Modules

Code:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

blacklist.conf

Code:

blacklist radeon
blacklist nouveau
blacklist nvidia

vfio.conf ( ye it show up 4 ids)

Code:

options vfio-pci ids=1002:73bf,1002:ab28,1002:73a6,1002:73a6 disable_vga=1

like a said the system see the GPU but it wont work, even after install the drivers

I found different ways to try to solve error 43, but none of them worked in my case, so I'm looking for help from someone who can give me a better answer, so I don't end up messing up the system with wrong commands.

Thank already for help

leesteken · Oct 16, 2023

I see lots of little things wrong or outdated in your configuration, but with a RX6900 you should not really need most of it.

Malombrado said:

amd_iommu=on is nonsense because it is on by default on AMD.
video=vesafb:off,efifb:off has been wrong for a long time and even if you would write it correctly, it would not influence recent Proxmox.
nofb nomodeset probably does nothing for Proxmox as it only uses a console without graphics.
pcie_acs_override=downstream,multifunction makes Proxmox ignore your IOMMU groups, which is a security risk and you probably don't need it.
Are you sure your Proxmox uses GRUB? What is the output of cat /proc/cmdline?

Malombrado said:
100.conf
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=0ff"'

This does not make sense for AMD and probably not for NVidia for some time now.

Malombrado said:
cpu: host,hidden=1, flags=+pcid

This is not useful for AMD and might confuse their drivers.

Malombrado said:
hostpcio: 0000:03:00,pcie=1,x-vga=1

hostpcio is wrong and should be hostpci0.
Primary GPU settings is for NVidia and should not be used for AMD.

Malombrado said:
machine: pc-g35-8.0

g35 is wrong, it should be q35

Malombrado said:
vga: none

This is right and should force output to the GPU and a physical display connected to it.

Malombrado said:
Modules

Code:

vfio vfio_iommu_type1 vfio_pci vfio_virqfd

vfio_virfd no longer exists in Proxmox 8.

Malombrado said:
blacklist.conf

Code:

blacklist radeon blacklist nouveau blacklist nvidia

This makes no sense for a GPU that uses amdgpu and should not be nessecary.

Malombrado said:
vfio.conf ( ye it show up 4 ids)

Code:

options vfio-pci ids=1002:73bf,1002:ab28,1002:73a6,1002:73a6 disable_vga=1

Try not doing early binding and just let amdgpu take control of the GPU before starting the VM.

Malombrado said:
I found different ways to try to solve error 43, but none of them worked in my case, so I'm looking for help from someone who can give me a better answer, so I don't end up messing up the system with wrong commands.

Is the RX6900 the only GPU in your system?

Try these steps:
Don't early bind the GPU functions to vfio-pci (and update-initramfs -u and reboot).
Run echo 0 | tee /sys/class/vtconsole/vtcon*/bind >/dev/null on the Proxmox host before starting the VM (to release the console from the GPU).
Boot your VM with an Ubuntu Live installer ISO (but don't install Ubuntu) to see if passthrough works without the complications of Windows (and you can stop/reset without damaging your VM filesystem).

PS: Can't you SSH into your Proxmox and copy text instead of typing it and making lots of little mistakes?

Tigo · Oct 16, 2023

Try the following this guide, with the following tidbits

- Bios settings. Disable Resizable bar. This would be somewhere in your PCIexpress settings. Also, if your motherboard has a built-in GC, and options on whether to select the inbuilt GC or the 6900xt (External - change the configuration to select the 6900xt).

- Grub file
"GRUB_CMDLINE_LINUX="quiet intel_iommu=on iommu=pt nomodeset initcall_blacklist=sysfb_init" works fine
"GRUB_CMDLINE_LINUX="quiet intel_iommu=on iommu=pt" - works fine

- modules
vfio
vfio_iommu_type1
vfio_pci

- (skip the vendor reset), - that just gave me errors.

- vfio.conf
options vfio-pci ids=(your GC ID), (your GC-audio ID) disable vga=1 *
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci

*you can also remove disable vga=1. It's unnecessary as its the display is set to non in the VM hardware.

- blacklist.conf
blacklist radeon
blacklist amdgpu

At the VM Hardware settings - set the display to none. Start the machine, and wait for a minute or so, till it boots. Passthrough should be successful. I had mapped the mouse and keyboard prior to starting up the VM.

I was stuck all day with trying to get the 6900xt to work, and in the end, it was just disabling the resizable bar that got me across. The guide above is excellent. Read it carefully, apply what's needed only, and not the whole thing.
Good luck

Malombrado · Oct 16, 2023

Thanks guys, I followed your @leesteken tips and part your @Tigo guide and I managed to make it work easily. I believe my problem was trying several outdated and even more complicated guides, I didn't even need to blacklist the drivers or bindings and the system worked perfectly.

Malombrado · Oct 17, 2023

I don't know if it would be worth opening another topic, but I'll continue on this one to even have knowledge about the configuration already carried out.

@leesteken regarding the system's GPUs I forgot to mention the CPU AMD 7950x3d, it has integrated graphics, if I'm not mistaken it's called Raphael:

Code:

18:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
18:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
18:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
        Subsystem: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
18:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
18:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
18:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]

I'm thinking about keeping an iGPU VM for work and keeping a 6900xt VM for gaming

UPDATE:
Today i tried to make teh iGPU passtrough, probably i'll need some more config to make the Video Output on monitor, but i still don't know how.
the conf made:

iGPU VM

Code:

bios: ovmf
boot: order=ide2;scsi
cores: 4
cpu: host
efidisk0: local-lvm:vm-101-disk-0,efitype=4m, pre-enrolled-keys=1,size=4M
hostpci0: 0000:18:00.0,pcie=1
hostpci1: 0000:18:00.1,pcie=1
ide2: local:iso/virtio-win-O.1.240.iso,media=cdrom, size=612812K
machine: pc-g35-8.0
memory: 16384
meta: creation-gemu=8.0.2,ctime=1697543232
name: Workstation
net0: virtio=EE:6D:C7:96:28:8A,bridge=vmbr0, firewall=1
numa: 0
ostype: winll
scsi0: local-lvm:vm-101-disk-1,iothread=1,size=80G
Scsihw: virtio-scsi-single
smbiosl: uuid=858da277-96f3-4e60-b58a-G6fddob336f6
sockets: 4
tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 4bfb3f28-8803-4987-8cc8-aa3f3d5af5b7

I didn't go passthrough all the functions because it just made my system halt.
There's some tests shows this one make the system halt (18:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7] )

I only tried VGA and HDMI audio and the system was still working. I installed Windows normally and it found the video driver, but adrenaline itself did not open, and the machine does not return video on the monitor even with VGA none, probably this error cod 43 again, probably because the all functions was disabled.

I was thinking that I need to prevent proxmox from accessing an IGPU so that it is available to the system, but i'd no lucky on that, the system still halt with all functions on.

UPDATE2: After some study and research just made it work.. Rly thanks guys i'm ready to start my build with full passtrought the 6900XT and IGPU 7950x3d both on 3 monitors

Tigo · Oct 18, 2023

Malombrado said:
I don't know if it would be worth opening another topic, but I'll continue on this one to even have knowledge about the configuration already carried out.

@leesteken regarding the system's GPUs I forgot to mention the CPU AMD 7950x3d, it has integrated graphics, if I'm not mistaken it's called Raphael:

Code:

18:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9) 18:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640] 18:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649] Subsystem: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649] 18:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6] 18:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7] 18:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]

I'm thinking about keeping an iGPU VM for work and keeping a 6900xt VM for gaming

UPDATE:
Today i tried to make teh iGPU passtrough, probably i'll need some more config to make the Video Output on monitor, but i still don't know how.
the conf made:

iGPU VM

Code:

bios: ovmf boot: order=ide2;scsi cores: 4 cpu: host efidisk0: local-lvm:vm-101-disk-0,efitype=4m, pre-enrolled-keys=1,size=4M hostpci0: 0000:18:00.0,pcie=1 hostpci1: 0000:18:00.1,pcie=1 ide2: local:iso/virtio-win-O.1.240.iso,media=cdrom, size=612812K machine: pc-g35-8.0 memory: 16384 meta: creation-gemu=8.0.2,ctime=1697543232 name: Workstation net0: virtio=EE:6D:C7:96:28:8A,bridge=vmbr0, firewall=1 numa: 0 ostype: winll scsi0: local-lvm:vm-101-disk-1,iothread=1,size=80G Scsihw: virtio-scsi-single smbiosl: uuid=858da277-96f3-4e60-b58a-G6fddob336f6 sockets: 4 tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0 vga: none vmgenid: 4bfb3f28-8803-4987-8cc8-aa3f3d5af5b7

I didn't go passthrough all the functions because it just made my system halt.
There's some tests shows this one make the system halt (18:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7] )

I only tried VGA and HDMI audio and the system was still working. I installed Windows normally and it found the video driver, but adrenaline itself did not open, and the machine does not return video on the monitor even with VGA none, probably this error cod 43 again, probably because the all functions was disabled.

View attachment 56773

I was thinking that I need to prevent proxmox from accessing an IGPU so that it is available to the system, but i'd no lucky on that, the system still halt with all functions on.

UPDATE2: After some study and research just made it work.. Rly thanks guys i'm ready to start my build with full passtrought the 6900XT and IGPU 7950x3d both on 3 monitors

View attachment 56782

how did you make it work? please share configs.... thanks

encrypted_ · Oct 30, 2023

+1 share config please

leesteken · Oct 30, 2023

Tigo said:
how did you make it work? please share configs.... thanks

encrypted_ said:
+1 share config please

This user already shared his config in this thread:
https://forum.proxmox.com/threads/passthrough-on-6900xt-and-igpu-7950x3d.134972/post-596804
https://forum.proxmox.com/threads/passthrough-on-6900xt-and-igpu-7950x3d.134972/post-597278
What more config do you want? How do you think it will help you? What problems are you running into?

encrypted_ · Oct 31, 2023

I'm stuck with Error 43 and the Adrenalin Error. Exactly like those screenshots.
I had checked both threads and followed along with the settings but haven't found a way to get rid of it.

103.conf

Code:

bios: seabios
boot: order=virtio0;net0;ide0
cores: 4
cpu: host,hidden=1,flags=+pcid
hostpci0: 0000:06:00.0,pcie=1,rombar=0,romfile=vbios_1636.dat
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
machine: pc-q35-6.2
memory: 8192
meta: creation-qemu=7.2.0,ctime=1698553568
name: win10-igpu
net0: virtio=E6:48:0A:D8:23:10,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=b725946b-3fd3-4f8e-aff2-ceb78533f4b0
sockets: 1
vga: none
virtio0: local-lvm:vm-103-disk-0,iothread=1,size=32G
vmgenid: a9ab640a-43de-409f-bd32-65d7c7d4c239

dmesg | grep AMD

Code:

[    0.000000]   AMD AuthenticAMD
[    0.006006] RAMDISK: [mem 0x30a77000-0x34532fff]
[    0.006030] ACPI: IVRS 0x00000000BA582000 0000D0 (v02 AMD    AmdTable 00000001 AMD  00000001)
[    0.006032] ACPI: SSDT 0x00000000BA57A000 00723C (v02 AMD    MYRTLE   00000002 MSFT 04000000)
[    0.006035] ACPI: SSDT 0x00000000BA576000 003AD2 (v01 AMD    AMD AOD  00000001 INTL 20120913)
[    0.006056] ACPI: SSDT 0x00000000BA565000 002808 (v02 AMD    AmdTable 00000001 AMD  00000001)
[    0.006058] ACPI: CRAT 0x00000000BA564000 000BA8 (v01 AMD    AmdTable 00000001 AMD  00000001)
[    0.006061] ACPI: CDIT 0x00000000BA563000 000029 (v01 AMD    AmdTable 00000001 AMD  00000001)
[    0.006063] ACPI: SSDT 0x00000000BA55F000 003802 (v01 AMD    MYRTLE   00000001 INTL 20120913)
[    0.006066] ACPI: SSDT 0x00000000BA55E000 0000BF (v01 AMD    AmdTable 00001000 INTL 20120913)
[    0.006074] ACPI: SSDT 0x00000000BA55B000 00051B (v01 AMD    MYRTLERN 00000001 INTL 20120913)
[    0.006076] ACPI: SSDT 0x00000000BA559000 0010AF (v01 AMD    MYRTLE   00000001 INTL 20120913)
[    0.286500] smpboot: CPU0: AMD Ryzen 5 PRO 4650G with Radeon Graphics (family: 0x17, model: 0x60, stepping: 0x1)
[    0.286660] Performance Events: Fam17h+ core perfctr, AMD PMU driver.
[    0.369603] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.370209] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.370210] AMD-Vi: Extended features (0x206d73ef22254ade): PPR X2APIC NX GT IA GA PC GA_vAPIC
[    0.370214] AMD-Vi: Interrupt remapping enabled
[    0.370214] AMD-Vi: Virtual APIC enabled
[    0.370214] AMD-Vi: X2APIC enabled
[    0.690111] perf: AMD IBS detected (0x000003ff)
[    0.690129] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    6.253254] AMD64 EDAC driver v3.5.0

encrypted_ · Oct 31, 2023

Interesting updates.
I managed to make it work, without error 43, after rebooting the host with the following configuration:

103.conf

Code:

bios: seabios
boot: order=virtio0;net0;ide0
cores: 4
cpu: host,hidden=1
hostpci0: 0000:06:00.0,pcie=1,romfile=vbios_1636.dat,x-vga=1
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
machine: pc-q35-6.2
memory: 8192
meta: creation-qemu=7.2.0,ctime=1698553568
name: win10-igpu
net0: virtio=E6:48:0A:D8:23:10,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=b725946b-3fd3-4f8e-aff2-ceb78533f4b0
sockets: 1
vga: none
virtio0: local-lvm:vm-103-disk-0,iothread=1,size=32G
vmgenid: a9ab640a-43de-409f-bd32-65d7c7d4c239

But after powering down, it won't turn on again.
I can start the VM, it shows as running but I can't RDP and it even shows in the DHCP map as online.

If I reboot, I can start it once with the iGPU but never again. Only adding a display which then breaks the igpu driver in Windows.

leesteken · Oct 31, 2023

encrypted_ said:

Interesting updates.
I managed to make it work, without error 43, after rebooting the host with the following configuration:

103.conf

Code:

bios: seabios
boot: order=virtio0;net0;ide0
cores: 4
cpu: host,hidden=1
hostpci0: 0000:06:00.0,pcie=1,romfile=vbios_1636.dat,x-vga=1
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
machine: pc-q35-6.2
memory: 8192
meta: creation-qemu=7.2.0,ctime=1698553568
name: win10-igpu
net0: virtio=E6:48:0A:D8:23:10,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=b725946b-3fd3-4f8e-aff2-ceb78533f4b0
sockets: 1
vga: none
virtio0: local-lvm:vm-103-disk-0,iothread=1,size=32G
vmgenid: a9ab640a-43de-409f-bd32-65d7c7d4c239

Primary GPU (x-vga=1) and hidden=1 is for NVidia GPUs and you probably don't need then.

encrypted_ said:
But after powering down, it won't turn on again.
I can start the VM, it shows as running but I can't RDP and it even shows in the DHCP map as online.

If I reboot, I can start it once with the iGPU but never again. Only adding a display which then breaks the igpu driver in Windows.

This is probably because the integrated GPU does not reset properly and can therefore only used once. It's not uncommon but I fear this GPU is not supported by vendor-reset.
Maybe you can find a work-around from someone on the internet?

encrypted_ · Oct 31, 2023

That's really where I am. Tested vendor-reset yesterday without success. Posted here hoping someone will find this message someday, with a possible solution.

I tested this (from here) but did not work also:
- before starting the vm, gpu hook/reset (I read about this in another forum post here)
echo 1 > /sys/bus/pci/devices/0000\:06\:00.0/remove
echo 1 > /sys/bus/pci/rescan

Search

Search

Passthrough on 6900xt and iGPU 7950x3d

Malombrado

New Member

leesteken

Distinguished Member

Tigo

New Member

Malombrado

New Member

Malombrado

New Member

Tigo

New Member

encrypted_

New Member

leesteken

Distinguished Member

encrypted_

New Member

encrypted_

New Member

leesteken

Distinguished Member

encrypted_

New Member