Problem Error 43 on AMD RX 6700XT

[Drthrax74]

Hi,

I need help for fixed error 43 on Windows 10 with my RX 6700XT.

Grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on initcall_blacklist=sysfb_init iommu=pt"

Modules:

echo "vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
amd_iommu=on" > /etc/modules

Dmesg

dmesg | grep -e DMAR -e IOMMU
[    0.675333] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.680045] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.687484] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

dmesg | grep 'remapping'
[    0.445775] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.680051] AMD-Vi: Interrupt remapping enabled

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

Groups:

find /sys/kernel/iommu_groups/ -type l | sort -n | grep "25\|26"
/sys/kernel/iommu_groups/25/devices/0000:0b:00.0
/sys/kernel/iommu_groups/26/devices/0000:0b:00.1

GPU:

lspci -n -s 0b:00
0b:00.0 0300: 1002:73df
0b:00.1 0403: 1002:ab28

Infos:
- 0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [1002:73df]
- 0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]

310.conf:

cpu: host,hidden=1,flags=+pcid
machine: q35
hostpci0: 0b:00.0,pcie=1,x-vga=on

VFIO: (OVMF)

echo "options vfio-pci ids=1002:73df,1002:ab28 disable_vga=1" > /etc/modprobe.d/vfio.conf

Blacklist:

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Note:
- GPU: https://forum.level1techs.com/t/radeon-rx-6950-xt-reset-bug-question/194617/6
- Autologon: https://learn.microsoft.com/fr-fr/t...er-profiles-and-logon/turn-on-automatic-logon

 

[leesteken]

Don't use Primary GPU (x-vga=on) for AMD GPUs, it's intended for NVidia GPUs. Make sure to passthrough the whole GPU with All Functions (0b:00) instead of only the first function (0b:00.0). Double check with cat /proc/cmdline if initcall_blacklist=sysfb_init is active when doing passthrough of the boot (or only) GPU.

There is also no need to hide (hidden=1) with AMD GPUs. And amd_iommu=on does nothing because it is on by default. You should not need iommu=pt nor flags=+pcid nor options vfio_iommu_type1 allow_unsafe_interrupts=1 and you should not put amd_iommu=on in /etc/modules (did you not get an error when running update-initramfs -u?).

Test if passthrough is working by booting the VM with a recent Ubuntu Live installer (without installing it): it should show output on a physical display connected to the GPU.

EDIT: There is a list on reddit with 6700XT cards that don't work with passthrough.

 

[hannibal1969]

@leesteken, found already numerous good post from you here instructing people how to passthrough their AMD GPUs. Your above "list on reddit" is not working. Can you update the link please?

Currently I'm learning my first small steps with Proxmox on a mini PC and I have successfully host pfSense / XPenology / Home Assistant and a Kubernetes Cluster. On my main PC I run a bare metal macOS Ventura with dual boot with Windows 11 too but I see a lot of people moving to Proxmox for this as well. For me this would make only sense If I can virtualise both windows (for gaming) and macOS (for my daily work) on the same GPU, Same multimonitor set, same mouse and keyboard set. Is this even possible with a AMD RX6800XT?

 

[leesteken]

@leesteken, found already numerous good post from you here instructing people how to passthrough their AMD GPUs. Your above "list on reddit" is not working. Can you update the link please?

Fixed it to https://www.reddit.com/r/VFIO/comments/tq9j5v/need_help_compiling_a_list_of_amd_6000_series/.

Currently I'm learning my first small steps with Proxmox on a mini PC and I have successfully host pfSense / XPenology / Home Assistant and a Kubernetes Cluster. On my main PC I run a bare metal macOS Ventura with dual boot with Windows 11 too but I see a lot of people moving to Proxmox for this as well. For me this would make only sense If I can virtualise both windows (for gaming) and macOS (for my daily work) on the same GPU, Same multimonitor set, same mouse and keyboard set. Is this even possible with a AMD RX6800XT?

Linux and Windows with a single display (two inputs) and a USB 2.0 switch works for me with a AMD 6950XT. It also worked with a RX570 before with vendor-reset.

 

[hannibal1969]

Fixed it to https://www.reddit.com/r/VFIO/comments/tq9j5v/need_help_compiling_a_list_of_amd_6000_series/.

Linux and Windows with a single display (two inputs) and a USB 2.0 switch works for me with a AMD 6950XT. It also worked with a RX570 before with vendor-reset.

I think you mean 2 outputs from the GPU ;-) Assigning a dedicated Monitor (permanently) to one OS is for me a no go. I understand that you can see and use both OS-es simultaneously but not required for me. Is there a way to assign both outputs to one OS if only one OS is active of the two?

 

[leesteken]

I think you mean 2 outputs from the GPU ;-) Assigning a dedicated Monitor (permanently) to one OS is for me a no go.

That's not what I meant and trust me; I know how I did my setup. I have two GPUs (each VM has one) and both are connected to the same display (which can switch between two inputs).

I understand that you can see and use both OS-es simultaneously but not required for me. Is there a way to assign both outputs to one OS if only one OS is active of the two?

I use ddcutil to switch my display from one input to the other (so from one GPU to the other GPU) when the USB keyboard switches from one VM to the other VM using udev-rules.
Maybe you can do something similar and switch both displays to their other inputs? Or get a fancy KVM switch that can output to two displays (from four GPU inputs).

 

[aldariz]

I had a similar issue after updating Proxmox to the latest stable release. Have you checked your BIOS setting are correct

https://pve.proxmox.com/wiki/PCI_Passthrough#GPU_OVMF_PCI_Passthrough_.28recommended.29
BIOS Options
"Resizable BAR'/'Smart Access Memory': Some AMD GPUs (Vega and up) experience 'Code 43' in Windows guests if this is enabled on the host. It's not supported in VMs either way (yet), so the recommended setting is 'off'."

 

[hannibal1969]

That's not what I meant and trust me; I know how I did my setup. I have two GPUs (each VM has one) and both are connected to the same display (which can switch between two inputs).

I use ddcutil to switch my display from one input to the other (so from one GPU to the other GPU) when the USB keyboard switches from one VM to the other VM using udev-rules.
Maybe you can do something similar and switch both displays to their other inputs? Or get a fancy KVM switch that can output to two displays (from four GPU inputs).

Ah indeed but if it requires 2 GPUs to instant swap between two 2 OS-es its a nog go for me. It's not fitting in my PC and costly, although for macOS Ventura my current RX6800XT is somewhat overkill, a RX580 or so is more then enough.

 

[leesteken]

Ah indeed but if it requires 2 GPUs to instant swap between two 2 OS-es its a nog go for me. It's not fitting in my PC and costly, although for macOS Ventura my current RX6800XT is somewhat overkill, a RX580 or so is more then enough.

RIght, I misunderstood. You can have one VM use the whole GPU (with all its outputs) at any one time. You cannot have one output with one VM and the other with another VM at the same time. PCIe passthrough is all or nothing. There are other ways like VirGL and mdev to share some types of GPU but they cannot output to a physical display.

 

[hannibal1969]

What could work is possibly an OS installed like linux Mint or so in Proxmox with PCI passthrough to the iGPU. This OS is also booted by default on monitor 1. From there you have 2 boot options with 2 VMs. 1 VM with PCI passtrough the RX6800XT for Windows 11 with monitor 2 and 3, the other option passthrough the RX6800XT to macOS also for monitor 2 and 3 but requires the Wndows 11 VM to be stopped and vice versa.

... and pops up the idea ... maybe the iGPU is good enough for macOS and assign dedicated the RX6800XT to the windows VM ... then I only need a KVM switch.

 

[Drthrax74]

I found solution for my problem.


Kernel:

6.2.9-1-pve (Module not found: vfio_virqfd)

lspci -nn -k | grep VGA -A3 | grep "VGA\|driver"
# 0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [1002:73df] (rev c1)
# Kernel driver in use: amdgpu


lspci -s 0b: -k -nn | grep -i "audio\|vga" -A2 | grep -iv "sub\|--"
# 0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [1002:73df] (rev c1)
# Kernel driver in use: amdgpu
#
# 0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
# Kernel driver in use: amdgpu

find /sys/kernel/iommu_groups/ -type l;
# /sys/kernel/iommu_groups/25/devices/0000:0b:00.0
# /sys/kernel/iommu_groups/26/devices/0000:0b:00.1

find /sys/kernel/iommu_groups/ -type l | grep "25\|26";
# /sys/kernel/iommu_groups/25/devices/0000:0b:00.0
# /sys/kernel/iommu_groups/26/devices/0000:0b:00.1

# find /usr/lib/modules -type f -name "vfio*" -o -name "vfio_iommu_type1*" -o -name "vfio_pci*" -o -name "vfio_virqfd*" 2>/dev/null | grep $(uname -r)
/usr/lib/modules/6.2.9-1-pve/kernel/drivers/vfio/vfio_iommu_type1.ko
/usr/lib/modules/6.2.9-1-pve/kernel/drivers/vfio/pci/vfio-pci.ko
/usr/lib/modules/6.2.9-1-pve/kernel/drivers/vfio/pci/vfio-pci-core.ko
/usr/lib/modules/6.2.9-1-pve/kernel/drivers/vfio/vfio.ko

Module not found: vfio_virqfd

GRUB

sed -i -e 's/quiet/quiet amd_iommu=on initcall_blacklist=sysfb_init/g' /etc/default/grub;

Modules

echo "vfio
vfio_iommu_type1
vfio_pci" > /etc/modules;

KVM:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf;
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf;

Blacklist

echo "# GPU AMD" > /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist ati" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist amdgpu" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist radeon" >> /etc/modprobe.d/pve-blacklist.conf;
echo "" >> /etc/modprobe.d/pve-blacklist.conf;
echo "# GPU Nvidia" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist nvidia" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist nvidiafb" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist snd_hda_intel" >> /etc/modprobe.d/pve-blacklist.conf;
echo "" >> /etc/modprobe.d/pve-blacklist.conf;
echo "# Manette Xbox" >> /etc/modprobe.d/pve-blacklist.conf;
echo "blacklist xpad" >> /etc/modprobe.d/pve-blacklist.conf;

Interrupt Remmaping

clear;
echo "###########################################################
# Nvidia: GTX 1060
# options vfio-pci ids=10de:1c02,10de:10f1 disable_vga=1
###########################################################
# Nvidia: RTX 3060
# options vfio-pci ids=10de:2503,10de:228e disable_vga=1
###########################################################
# AMD: RX 6700 XT
options vfio-pci ids=1002:73df,1002:ab28 disable_vga=1
###########################################################
# Broadcom: BCM4352 802.11ac
# options vfio-pci ids=14e4:43b1
###########################################################" > /etc/modprobe.d/vfio.conf;

nano /etc/modprobe.d/vfio.conf;

Update Boot:

update-initramfs -u;
update-grub;
update-grub2;
reboot;

Result:

0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [1002:73df] (rev c1)
Kernel driver in use: vfio-pci
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
Kernel driver in use: vfio-pci

 

[levtol]

s

I found solution for my problem.

Thank you very much for highlighting this moments

# Kernel driver in use: amdgpu
update-grub2;
Kernel driver in use: vfio-pci

, after "update-grub2" error 43 have gone for my 6800

 

[Drthrax74]

Can you write action and more information ?