[SOLVED] AMD machine only gets IOMMU enabled when setting intel_iommu=on... false positive?

[rhobinn]

I have set up GPU passthrough with a few computers (all intel) but I have one that eludes me, and I don't have a clue what's happening:
I'm trying to get IOMMU enabled and keep failing

First, this is my hardware:

AMD ryzen 3900x
Gigabyte 570x gaming X
GeForce RTX 3080Ti GAMING X TRIO

I'm using Proxomox 7.2-3
Linux 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200)


My bios is updated to the last version (F37d), and have this settings (have tried them enabling and disabling each one of them one by one):
IOMMU: Enabled
CSM: Disabled
Above 4G decoding: Enabled
Resizable BAR: Disabled
ACS: Enabled

I have another GPU (cuadro p620) but that's only for monitor output... and the bios is set to output on this one
the 3080 card seems to be on its own IOMMU Group (25)

lspci -nnv | grep VGA
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P620] [10de:1cb6] (rev a1) (prog-if 00 [VGA controller]) 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2208] (rev a1) (prog-if 00 [VGA controller])

lspci -s 0a:00 -n
0a:00.0 0300: 10de:2208 (rev a1) 0a:00.1 0403: 10de:1aef (rev a1)

find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.2 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/4/devices/0000:00:03.1 /sys/kernel/iommu_groups/5/devices/0000:00:04.0 /sys/kernel/iommu_groups/6/devices/0000:00:05.0 /sys/kernel/iommu_groups/7/devices/0000:00:07.0 /sys/kernel/iommu_groups/8/devices/0000:00:07.1 /sys/kernel/iommu_groups/9/devices/0000:00:08.0 /sys/kernel/iommu_groups/10/devices/0000:00:08.1 /sys/kernel/iommu_groups/11/devices/0000:00:14.3 /sys/kernel/iommu_groups/11/devices/0000:00:14.0 /sys/kernel/iommu_groups/12/devices/0000:00:18.3 /sys/kernel/iommu_groups/12/devices/0000:00:18.1 /sys/kernel/iommu_groups/12/devices/0000:00:18.6 /sys/kernel/iommu_groups/12/devices/0000:00:18.4 /sys/kernel/iommu_groups/12/devices/0000:00:18.2 /sys/kernel/iommu_groups/12/devices/0000:00:18.0 /sys/kernel/iommu_groups/12/devices/0000:00:18.7 /sys/kernel/iommu_groups/12/devices/0000:00:18.5 /sys/kernel/iommu_groups/13/devices/0000:01:00.0 /sys/kernel/iommu_groups/14/devices/0000:02:01.0 /sys/kernel/iommu_groups/15/devices/0000:02:02.0 /sys/kernel/iommu_groups/16/devices/0000:02:03.0 /sys/kernel/iommu_groups/17/devices/0000:02:04.0 /sys/kernel/iommu_groups/18/devices/0000:07:00.0 /sys/kernel/iommu_groups/18/devices/0000:02:08.0 /sys/kernel/iommu_groups/18/devices/0000:07:00.3 /sys/kernel/iommu_groups/18/devices/0000:07:00.1 /sys/kernel/iommu_groups/19/devices/0000:08:00.0 /sys/kernel/iommu_groups/19/devices/0000:02:09.0 /sys/kernel/iommu_groups/20/devices/0000:09:00.0 /sys/kernel/iommu_groups/20/devices/0000:02:0a.0 /sys/kernel/iommu_groups/21/devices/0000:03:00.0 /sys/kernel/iommu_groups/22/devices/0000:04:00.1 /sys/kernel/iommu_groups/22/devices/0000:04:00.0 /sys/kernel/iommu_groups/23/devices/0000:05:00.0 /sys/kernel/iommu_groups/24/devices/0000:06:00.0 /sys/kernel/iommu_groups/25/devices/0000:0a:00.0 /sys/kernel/iommu_groups/25/devices/0000:0a:00.1 /sys/kernel/iommu_groups/26/devices/0000:0b:00.0 /sys/kernel/iommu_groups/27/devices/0000:0c:00.0 /sys/kernel/iommu_groups/28/devices/0000:0c:00.1 /sys/kernel/iommu_groups/29/devices/0000:0c:00.3 /sys/kernel/iommu_groups/30/devices/0000:0c:00.4

My sistem supports remapping

dmesg | grep 'remapping'
[ 0.370033] x2apic: IRQ remapping doesn't support X2APIC mode [ 0.635832] AMD-Vi: Interrupt remapping enabled

I added to /etc/modules the modules suggested
vfio vfio_iommu_type1 vfio_pci vfio_virqfd

I have tried different things in the fail /etc/default/grub
but if I put

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on"

I get this from dmesg | grep -e DMAR -e IOMMU
[ 0.629125] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported [ 0.634719] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.635124] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

I have tried different options:
-quiet
-vfio-pci.ids=10de:1e81,10de:10f8,10de:1ad8,10de:1ad9
-modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
-
and every time I get exactly the same message

only when I add:
-pcie_acs_override=downstream,multifunction
it adds the line to the dmesg results:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

but I cannot seem to be able to enable IOMMU

but...
by mistake I used
intel_iommu=on

and then I get

dmesg | grep -e DMAR -e IOMMU

[ 0.084082] DMAR: IOMMU enabled [ 0.627289] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported [ 0.632809] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.633146] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

and I don't know if that is actually correct...
why does this happen? it is not an intel processor... why does IOMMU turns on when I set intel_iommu=on??
is it a false positive?
or that's correct and I should use intel_iommu=on?

what should I do?

to test the GPU on the VM...
im gonna be running a simulation software GROMACS, and installing it and compiling it is long... so I haven't gotten to it
I will try it later to see if with the intel ON setting works...

but meanwhile, what do you think is happening? why IOMMU only "works" when I set it to intel instead of AMD?? what should I do?

 

[leesteken]

amd_iommu is on by default.

Invalid parameters are ignored, therefore amd_iommu=on is ignored (because it is not defined) as well as intel_iommu=on (because it does not apply to your system). Resulting in amd_iommu being enabled just as it would when you don't add any parameters.
I also think you are using the check for IOMMU being enabled for Intel systems, which does not apply.
You have several IOMMU groups (0 upto 30), so IOMMU is definitely enabled (which it is by default) and I don't see any problems.

If you are having problems with a VM with passthrough, it's not because IOMMU is not enabled but probably because the GPU is used during boot and by Proxmox. The various work-arounds are in this thread from several months ago.

 

[rhobinn]

solved here
https://forum.proxmox.com/threads/is-gpu-passthrough-working.115919/