Is IOMMU enabled on my System

[mfreudenberg]

Hi,

I've installed Proxmox VE 7.4 on my old Gaming-RIG. I have followed the instructions on this page, to enable IOMMU. When i run the "check command" `dmesg | grep -e DMAR -e IOMMU -e AMD-Vi` i get the following output:

root@pve-g:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.159493] AGP: Please enable the IOMMU option in the BIOS setup
[    0.765928] PCI-DMA: using GART IOMMU.
[    0.765930] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    5.291498] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

I'm not quite certain, that IOMMU is enabled on my system. Can anyone help me to ensure, that i have either IOMMU correctly enabled or if it's just not compatible.

According to this Wiki-Page i assume that IOMMU should be supported by my system. Still i can neither exclude nor confirm, that my system supports IOMMU.

below my System hardware:

- CPU: AMD FX-8320E Eight-Core Processor
- Mainboard: ASRock 980DE
- RAM: 24GB DDR3
- GPU: Sapphire Radeo RX470 8GB VRAM

Thanks and BR,
Michael

 

[leesteken]

If IOMMU is supported and (fully) enabled in the motherboard BIOS, then it is also enabled in Proxmox automatically.
I think you are used a test for Intel systems. What is the full output of cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? If there is only one group numbered * then it's not enabled (in BIOS), otherwise it is and you'll know which devices are in the same group (devices in a group cannot be split between VMs and/or the Proxmox host).

 

[mfreudenberg]

If IOMMU is supported and (fully) enabled in the motherboard BIOS, then it is also enabled in Proxmox automatically.
I think you are used a test for Intel systems. What is the full output of cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? If there is only one group numbered * then it's not enabled (in BIOS), otherwise it is and you'll know which devices are in the same group (devices in a group cannot be split between VMs and/or the Proxmox host).

Thanks for your reply. Here is what i get (a lot as it seems):

BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet
IOMMU group * 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RX790 Host Bridge [1002:5957]
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RD790 PCI to PCI bridge (external gfx0 port A) [1002:5978]
00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RD790 PCI to PCI bridge (PCI express gpp port D) [1002:597d]
00:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e]
00:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD790 PCI to PCI bridge (PCI express gpp port F) [1002:597f]
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391]
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:12.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0 USB OHCI1 Controller [1002:4398]
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:13.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0 USB OHCI1 Controller [1002:4398]
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 3c)
00:14.1 IDE interface [0101]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller [1002:439c]
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383]
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d]
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384]
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev cf)
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
02:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
03:00.0 USB controller [0c03]: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller [1b6f:7052]
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)

I guess, that means IOMMU is enabled?

 

[mfreudenberg]

Additional Question: If i want to passthrough my GPU, and It's the only GPU the host has, do i need a secondary GPU, or would that work? I guess i will not be able to direklty use the host with monitor, keyboard and mouse, right?

 

[leesteken]

Thanks for your reply. Here is what i get (a lot as it seems):

BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet
IOMMU group * 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RX790 Host Bridge [1002:5957]
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RD790 PCI to PCI bridge (external gfx0 port A) [1002:5978]
00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RX780/RD790 PCI to PCI bridge (PCI express gpp port D) [1002:597d]
00:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e]
00:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD790 PCI to PCI bridge (PCI express gpp port F) [1002:597f]
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391]
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:12.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0 USB OHCI1 Controller [1002:4398]
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:13.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0 USB OHCI1 Controller [1002:4398]
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 3c)
00:14.1 IDE interface [0101]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller [1002:439c]
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383]
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d]
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384]
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev cf)
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
02:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
03:00.0 USB controller [0c03]: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller [1b6f:7052]
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)

I guess, that means IOMMU is enabled?

No, there is only one group so you cannot passthrough anything and IOMMU is not enabled, sorry.

Additional Question: If i want to passthrough my GPU, and It's the only GPU the host has, do i need a secondary GPU, or would that work? I guess i will not be able to direklty use the host with monitor, keyboard and mouse, right?

You'll probably need this work-around when doing passthrough of the boot (or only) GPU.

But you'll need to find a way to enable IOMMU (AMD-Vi) in the motherboard BIOS. I don't know much about AMD before Ryzen, so I can't tell if it can be enabled. Maybe you can check the manual or ask ASRock? Maybe someone else here or elsewhere knows...

 

[chrcoluk]

There is a second way to enable IOMMU, isnt well known, but ASRock did a custom bios for me so I could have it working on a G chip. Until I next go in my bios I cant help on the setting (they eventually made it public, but not all of their boards).

I also have these 2 in my cmdline, but not sure if they required.

'amd_iommu=on iommu=pt'

 

[leesteken]

I also have these 2 in my cmdline, but not sure if they required.
'amd_iommu=on iommu=pt'

amd_iommu=on does nothing (and is even invalid and therefore ignored) because it is on by default. iommu=pt sets the default mapping to identity for devices that are not passed through, and is not necessary for almost everybody. I know that just about every (outdated) guide everywhere adds those two but nobody can explain why.

 

[jsterr]

amd_iommu=on does nothing (and is even invalid and therefore ignored) because it is on by default. iommu=pt sets the default mapping to identity for devices that are not passed through, and is not necessary for almost everybody. I know that just about every (outdated) guide everywhere adds those two but nobody can explain why.

There is a VERY VERY important reason to use those parameters (or at least the =pt) because on lots of network cards greater 10GBit/s you not even getting close to full network performance. Craziest thing I had was NVIDIA 100Gbit Network Card only giving 500 MBit without parameters but 93Gbit/s with. Thats a extreme case, but on other cards you also gain a few % more network speed.

 

[leesteken]

There is a VERY VERY important reason to use those parameters (or at least the =pt) because on lots of network cards greater 10GBit/s you not even getting close to full network performance. Craziest thing I had was NVIDIA 100Gbit Network Card only giving 500 MBit without parameters but 93Gbit/s with. Thats a extreme case, but on other cards you also gain a few % more network speed.

Thank you for the explanation of a use case for iommu=pt! I assume you use those high-end network devices on the Proxmox host (and not with passthrough)?

 

[mfreudenberg]

No, there is only one group so you cannot passthrough anything and IOMMU is not enabled, sorry.

Hey guys,

thanks for all the feedback!

So i checked, the BIOS and couldn't find any setting, that sounds similar to anything mentioned in the proxmox wiki (like AMD-vi). The only option i seem to have is: Secure Virtual Machine

You may also take a look into the manual if you can find the right setting. But searching for "IOMMU", did not return anything.

Giving a try in the proxmox web-ui gives me the confirmation, that IOMMU is not enabled :-(.




So my hope was to reuse my old gaming rig for VM's that can issue the GPU, but i guess i'll have to go "bare metal". Is there any other way to get a GPU passed through to a guest? My goal was to setup "stable diffusion" server for generation of AI images.

Nevertheless,
Thanks a lot!

 

[jsterr]

Thank you for the explanation of a use case for iommu=pt! I assume you use those high-end network devices on the Proxmox host (and not with passthrough)?

Correct! So yeah inside a vm I did not have a usecase yet, but I guess if they cards wont work on the host, neither would they work with fullspeed with pci-passthrough (just a guess, might be wrong) But yeah its usually for high performance ceph network on the hypervisor host.

 

[chrcoluk]

amd_iommu=on does nothing (and is even invalid and therefore ignored) because it is on by default. iommu=pt sets the default mapping to identity for devices that are not passed through, and is not necessary for almost everybody. I know that just about every (outdated) guide everywhere adds those two but nobody can explain why.

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html?highlight=amd_iommu

As you said the on parameter isnt a valid option for it, so thanks for pointing that out. So my iommu works in its default state, 'pgtbl_v1' the default configuration.

Maybe I got the syntax from this post? who knows. https://forum.proxmox.com/threads/solved-https-pve-proxmox-com-wiki-pci_passthrough-pt-mode.102344/

I will keep the iommu=pt as its working well.

 

[rupert160]

amd_iommu=on does nothing (and is even invalid and therefore ignored) because it is on by default. iommu=pt sets the default mapping to identity for devices that are not passed through, and is not necessary for almost everybody. I know that just about every (outdated) guide everywhere adds those two but nobody can explain why.

Well noted @leesteken thanks.