[SOLVED] PCIe Passthrough - IOMMU, VT-d and setup done, but no groups

[jakeowaty]

Hello there!

I've probably been banging my head against this for the past few days and I am still stuck at the same problem.

So naturally I want to use PCIe passthrough on my Nvidia GPU on my homelab host. I've enabled VT-d and IOMMU in BIOS (and even reset it by removing CMOS battery to check if that would help, no dice).

Despite following guides to the T, I still get no output for:

find /sys/kernel/iommu_groups/ -type l

My /etc/default/grub looks like this:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nofb nomodeset initcall_bl>
GRUB_CMDLINE_LINUX=""

Which then I verify with:

root@homelab:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.8.12-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt nofb nomodeset initcall_blacklist=sysfb_init video=vesafbff,efifbff vfio-pci.ids=10de:2182,10de:1aeb disable_vga=1

root@homelab:~# lsmod | grep iommu
vfio_iommu_type1       45056  0
vfio                   65536  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                94208  1 vfio

root@homelab:~# dmesg |grep -e DMAR -e IOMMU -e AMD-Vi
[    0.060794] DMAR: IOMMU enabled

Only headway I've done is to get the following output when using the command:

root@homelab:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
03:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller [1969:e091] (rev 13

I am starting to think the hardware I'm using might just be not supporting the IOMMU properly, therefore the problem.

Would appreciate anyone helping me with this!

 

[leesteken]

intel_iommu=on is no longer needed since it is on by default since kernel version 6.8. video=vesafbff,efifbff has been invalid for a long time and does nothing on Proxmox.

If you don't have IOMMU groups then VT-d is not enabled by the motherboard BIOS or the motherboard or CPU does not support it.
Please double check your BIOS settings. I know you wrote that you did this already but it is still the most common cause of IOMMU not being enabled as people confuse VT-x and VT-d.
What your motherboard and CPU model?. Please double check them against the Intel website for VT-d support.

 

[jakeowaty]

intel_iommu=on is no longer needed since it is on by default since kernel version 6.8. video=vesafbff,efifbff has been invalid for a long time and does nothing on Proxmox.

That's what I get for following old guides, right?

Should I just remove the entire line from the /etc/default/grub? Since it doesn't matter, I'll try if that fixes things.

The machine is an older one - Intel I-7 4770k and MSI Z87-G45 Gaming. As far as I can see the CPU should support IOMMU, but motherboard might not. Although I for sure turned VT to D-mode (and its associated option turns to PEG automatically). I know it's on, since I saved and rebooted I get no video output to BIOS or server unless I take the GPU out.

Thank you so much for responding, I greatly appreciate it!

 

[leesteken]

The machine is an older one - Intel I-7 4770k and MSI Z87-G45 Gaming. As far as I can see the CPU should support IOMMU, but motherboard might not.

The motherboard probably supports it (because of the VT-d option in the BIOS) but your CPU does not support VT-d: https://ark.intel.com/content/www/u...-4770k-processor-8m-cache-up-to-3-90-ghz.html (bottom of the page Intel® Virtualization Technology for Directed I/O (VT-d) No)

 

[jakeowaty]

Gonna wrap this one up folks - CPU doesn't support VT-d. I've investigated options available and decided to abandon the project and stick with the pre-existing setup and use what I have.

Lesson for the future. Thanks for the help leesteken! It was very much appreciated.