[SOLVED] Need help on PCIe passthrough(intel iommu)!

alex01763

Member
Feb 28, 2023
12
1
8
Hi everyone

As you might have guessed, I'm having trouble with getting pcie passthrough to work. I think I've tried everything I can find on google and youtube, which were mostly the same. Here's my progress so far:

My /etc/default/grub line is: GRUB_CMDLINE_LINUX_DEFAULT="quiet nmi_watchdog=1 intel_iommu_on iommu=pt"
I tried various acs override configurations but to no avail.

When I run the dmsg command in the proxmox manual I get this:
Code:
root@node-main:~/Scripts# dmesg | grep -e DMAR -e IOMMU
[    0.009413] ACPI: DMAR 0x000000008A3EC828 0000A8 (v01 INTEL  EDK2     00000002      01000013)
[    0.009438] ACPI: Reserving DMAR table memory at [mem 0x8a3ec828-0x8a3ec8cf]
[    0.096874] DMAR: Host address width 39
[    0.096875] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.096880] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.096883] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.096886] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.096887] DMAR: RMRR base: 0x0000008aaca000 end: 0x0000008ad13fff
[    0.096889] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.096891] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.096892] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.096893] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.098903] DMAR-IR: Enabled IRQ remapping in x2apic mode

I found out on arch linux forum that the last line means vt-d is enabled, if I understand correctly.
But when I run this line I found in this forum:
Code:
root@node-main:~/Scripts# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU Group * 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:3e0f] (rev 08)
00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT1 [UHD Graphics 610] [8086:3e93]
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 [8086:a33c] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a303] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

I don't get any IOMMU groups. There aren't any third party gpus visible because I've sent my gtx 960 to get it fixed right now but still, I can't add any pcie hardware to my virtual machines because 'iommu is not enabled'.

I've spent too much time working on it now and I need your help. Thanks in advance!
 
If there are no multiple groups (only *) then IOMMU is not enabled. What is the output of cat /proc/cmdline? Maybe your Proxmox uses systemd-boot instead of GRUB?
I've checked that also, I think it uses grub, but I'm not a linux expert so I'm not a 100% sure. here's the output of cat

root@node-main:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.85-1-pve root=/dev/mapper/pve-root ro quiet nmi_watchdog=1 intel_iommu_on iommu=pt crashkernel=384M-:256M
 
I've checked that also, I think it uses grub, but I'm not a linux expert so I'm not a 100% sure. here's the output of cat

root@node-main:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.85-1-pve root=/dev/mapper/pve-root ro quiet nmi_watchdog=1 intel_iommu_on iommu=pt crashkernel=384M-:256M
I agree, it is using GRUB. However, the correct way to enable IOMMU on Intel is with intel_iommu=on and not what you have currently written in GRUB. The second underscore is a typing mistake and should be an equals sign.
 
  • Like
Reactions: alex01763
I agree, it is using GRUB. However, the correct way to enable IOMMU on Intel is with intel_iommu=on and not what you have currently written in GRUB. The second underscore is a typing mistake and should be an equals sign.
Unbelievable, how did I make such a mistake! Like you said, I changed intel_iommu_on to intel_iommu=on and it worked. Thank you so much!