Intel IOMMU - Dell Poweredge 1950iii

[pietroaretino]

I've been looking through the Proxmox forums for the past two days in regards to enabling IOMMU on my server.
I am trying to enable IOMMU in order to do a PCI passthrough of my Intel 3650 NIC.
I have another Dell Poweredge R510 with the the same NIC and version of Proxmox with no issues.

Here is my current info:
-PVE 8.0.4
-Linux 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z
-Dell Poweredge 1950iii
-BIOS Version 2.7.0
-VT is Capable and Enabled
-Execute Disable(XD) is Capable and Enabled
-2 x Intel(R) Xeon(R) CPU L5420 @ 2.50GHz Model 23 Stepping 6
-PERC 5/i hardware RAID Controller 4 x 1TB HDD in RAID10

Output of for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 5000X Chipset Memory Controller Hub [8086:25c0] (rev 12)
00:02.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 [8086:25e2] (rev 12)
00:03.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 [8086:25e3] (rev 12)
00:04.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 [8086:25f8] (rev 12)
00:05.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 [8086:25e5] (rev 12)
00:06.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 [8086:25f9] (rev 12)
00:07.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 [8086:25e7] (rev 12)
00:10.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers [8086:25f0] (rev 12)
00:10.1 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers [8086:25f0] (rev 12)
00:10.2 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers [8086:25f0] (rev 12)
00:11.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset Reserved Registers [8086:25f1] (rev 12)
00:13.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset Reserved Registers [8086:25f3] (rev 12)
00:15.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FBD Registers [8086:25f5] (rev 12)
00:16.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FBD Registers [8086:25f6] (rev 12)
00:1c.0 PCI bridge [0604]: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 [8086:2690] (rev 09)
00:1d.0 USB controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 [8086:2688] (rev 09)
00:1d.1 USB controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 [8086:2689] (rev 09)
00:1d.2 USB controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 [8086:268a] (rev 09)
00:1d.3 USB controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 [8086:268b] (rev 09)
00:1d.7 USB controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller [8086:268c] (rev 09)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev d9)
00:1f.0 ISA bridge [0601]: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller [8086:2670] (rev 09)
00:1f.2 IDE interface [0101]: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller [8086:2680] (rev 09)
01:00.0 PCI bridge [0604]: Intel Corporation 80333 Segment-A PCIe Express to PCI-X bridge [8086:0370]
01:00.2 PCI bridge [0604]: Intel Corporation 80333 Segment-B PCIe Express to PCI-X bridge [8086:0372]
02:0e.0 RAID bus controller [0104]: Dell PowerEdge Expandable RAID controller 5 [1028:0015]
04:00.0 PCI bridge [0604]: Broadcom EPB PCI-Express to PCI-X Bridge [1166:0103] (rev c3)
05:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5708 Gigabit Ethernet [14e4:164c] (rev 12)
06:00.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port [8086:3500] (rev 01)
06:00.3 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge [8086:350c] (rev 01)
07:00.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 [8086:3510] (rev 01)
07:01.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 [8086:3514] (rev 01)
08:00.0 PCI bridge [0604]: Broadcom EPB PCI-Express to PCI-X Bridge [1166:0103] (rev c3)
09:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5708 Gigabit Ethernet [14e4:164c] (rev 12)
0e:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
0e:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
0e:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
0e:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
10:0d.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 [1002:515e] (rev 02)

Output of dmesg | grep DMAR

DMAR: IOMMU enabled

However I know this output is not actually relevant and my dmesg should have something regarding Intel Virtualization Technology

When I grep dmesg for something along the lines of "Intel(R) Virtualization Technology for Directed I/O", I get nothing.
When I grep dmesg for "PCI-DMA" I get: [ 1.031625] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

Current GRUB configuration:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

I have added the following parameters to my kernel modules conf /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Here is the output of cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

Output of efibootmgr -v

EFI variables are not supported on this system.

So according to the Proxmox Manual I am definitely using GRUB

I have rebooted several times.

Not really sure what the problem could be, any ideas or tips would be much appreciated, thank you.

I will update this thread in regards to my troubleshooting should it prove useful for someone else in the future.

Cheers.