[SOLVED] PCI Passthrough issue

[dabaers]

Hey guys, second run at trying to get PCI passthrough working within proxmox,

Server - 2630L v4, supermicro x10srl-f, Quadro T400 and Intel 4x1gbe nic

GRUB
If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
#GRUD_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction v>
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

/etc/kernal/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

DMAR
[ 0.014584] ACPI: DMAR 0x0000000079863C50 0000BC (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[ 0.014641] ACPI: Reserving DMAR table memory at [mem 0x79863c50-0x79863d0b]
[ 0.395982] DMAR: Host address width 46
[ 0.395985] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[ 0.395993] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[ 0.395998] DMAR: RMRR base: 0x0000007ba18000 end: 0x0000007ba27fff
[ 0.396002] DMAR: ATSR flags: 0x0
[ 0.396005] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 0.396010] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 0
[ 0.396013] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0
[ 0.396017] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[ 0.396020] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.396326] DMAR-IR: Enabled IRQ remapping in x2apic mode

find /sys/kernel/iommu_groups/ -type l
(no return)

Bios level
Virtualization, SRIOV and X2apic have been enabled also above 4g decoding and efi oprom for both pci ports have been selected.

Error in Vm is states that NO IOMMU DETECTED; additionally, this board is stated to have C612 chipset (which is SR-IOV and VT-d) but proxmox reads c610 chipset in PCI selection menu.

I need help with this if anything can!

 

[leesteken]

Have a look at cat /proc/cmdline to double check that intel_iommu=on is actually active.
If find /sys/kernel/iommu_groups/ -type l is empty then IOMMU/VT-d is not (fully) enabled (in the BIOS and kernel) or not supported (by the hardware or the kernel).

 

[dabaers]

I would be really surprised if vt-d/iommu didn't work on this board and CPU as advertised, and it is enterprise gear.

cat /proc/cmdline

initrd=\EFI\proxmox\5.13.19-2-pve\initrd.img-5.13.19-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

 

[leesteken]

initrd=\EFI\proxmox\5.13.19-2-pve\initrd.img-5.13.19-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

As you can see, there is no intel_iommu=on in that line, therefore it is not enabled (in the kernel). It looks like you changed GRUB but Proxmox uses systemd-boot for your system.
Please add intel_iommu=on at the end of the first line in /etc/kernel/cmdline and run proxmox-boot-tool refresh before rebooting.

 

[dabaers]

leesteken, you are a god amongst men! I have iommu groups coming up now!!!!

 

[KarlensShklar]

Hey thanks for the super helpful reply. I'm not sure how I missed that thread. I haven't quite mastered the search function on here. I think I'll pass with him this time around. If i want any further guideline we will contact you here https://forum.proxmox.com/threads/pci-results-passthrough-issue.108176/.