Enable Passthrough Fails

[adresner]

Followed the steps here
https://pve.proxmox.com/wiki/PCI_Passthrough
Cross checked my steps here
https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/
BIOS is correct.

GRub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="quiet intel_iommu=on"

Modules
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Results
root@BKK-PX1:~# dmesg | grep -e DMAR -e IOMMU
[ 0.005290] ACPI: DMAR 0x000000006D84E000 000050 (v02 INTEL EDK2 00000002 01000013)
[ 0.005328] ACPI: Reserving DMAR table memory at [mem 0x6d84e000-0x6d84e04f]
[ 0.167910] DMAR: Host address width 39
[ 0.167911] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.167915] DMAR: dmar0: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[ 0.167918] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0
[ 0.167919] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.167920] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.169323] DMAR-IR: Enabled IRQ remapping in x2apic mode

Any suggestions?

 

[leesteken]

What is the output of cat /proc/cmdline? If intel_iommu=on is not there, then maybe your Proxmox does not use GRUB? What CPU and motherboard are you using?

EDIT: Nevermind, you got IOMMU working already.

 

[adresner]

Progress

Nano /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs
changed to
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

Any comments on the output?

root@BKK-PX1:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.005681] ACPI: DMAR 0x000000006D84E000 000050 (v02 INTEL EDK2 00000002 01000013)
[ 0.005715] ACPI: Reserving DMAR table memory at [mem 0x6d84e000-0x6d84e04f]
[ 0.070693] DMAR: IOMMU enabled
[ 0.168215] DMAR: Host address width 39
[ 0.168216] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.168219] DMAR: dmar0: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[ 0.168221] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0
[ 0.168222] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.168223] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.169625] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.451445] DMAR: No RMRR found
[ 0.451445] DMAR: No ATSR found
[ 0.451446] DMAR: No SATC found
[ 0.451447] DMAR: dmar0: Using Queued invalidation
[ 0.454158] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 11.258067] DMAR: DRHD: handling fault status reg 2
[ 11.258073] DMAR: [DMA Read NO_PASID] Request device [01:00.0] fault addr 0x0 [fault reason 0x06] PTE Read access is not set

root@BKK-PX1:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:01:00.0
/sys/kernel/iommu_groups/17/devices/0000:01:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:15.1
/sys/kernel/iommu_groups/7/devices/0000:00:15.2
/sys/kernel/iommu_groups/7/devices/0000:00:15.0
/sys/kernel/iommu_groups/25/devices/0000:08:00.3
/sys/kernel/iommu_groups/15/devices/0000:00:1d.0
/sys/kernel/iommu_groups/5/devices/0000:00:0e.0
/sys/kernel/iommu_groups/23/devices/0000:08:00.1
/sys/kernel/iommu_groups/13/devices/0000:00:1c.1
/sys/kernel/iommu_groups/3/devices/0000:00:06.0
/sys/kernel/iommu_groups/21/devices/0000:07:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:1b.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/18/devices/0000:02:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.3
/sys/kernel/iommu_groups/16/devices/0000:00:1f.0
/sys/kernel/iommu_groups/16/devices/0000:00:1f.5
/sys/kernel/iommu_groups/16/devices/0000:00:1f.4
/sys/kernel/iommu_groups/6/devices/0000:00:14.2
/sys/kernel/iommu_groups/6/devices/0000:00:14.0
/sys/kernel/iommu_groups/24/devices/0000:08:00.2
/sys/kernel/iommu_groups/14/devices/0000:00:1c.4
/sys/kernel/iommu_groups/4/devices/0000:00:0a.0
/sys/kernel/iommu_groups/22/devices/0000:08:00.0
/sys/kernel/iommu_groups/12/devices/0000:00:1c.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/20/devices/0000:06:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1a.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:03:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:17.0

 

[adresner]

What is the output of cat /proc/cmdline? If intel_iommu=on is not there, then maybe your Proxmox does not use GRUB? What CPU and motherboard are you using?

Ha! All along I thought I was using Grub? I guess I am not?
Pro WS W680-ACE IPMI
i9 13900k

root@BKK-PX1:~# efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0000,000F,0010,000B,000C,000D,000E,0002,0003,0004
Boot0000* Linux Boot Manager HD(2,GPT,232ac1bd-b37a-4bfa-bf2c-0e8b7efbddf1,0x800,0x100000)/File(\EFI\SYSTEMD\SYSTEMD-BOOTX64.EFI)
Boot0001* Linux Boot Manager HD(2,GPT,0e7a8476-f6c7-4ede-b992-61101010bcd0,0x800,0x100000)/File(\EFI\SYSTEMD\SYSTEMD-BOOTX64.EFI)
Boot0002* UEFI:CD/DVD Drive BBS(129,,0x0)
Boot0003* UEFI:Removable Device BBS(130,,0x0)
Boot0004* UEFI:Network Device BBS(131,,0x0)
Boot000B* UEFI: LAN1 PXE IPv4 Intel(R) Ethernet Controller I226-LM PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(c87f545af3bf,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000C* UEFI: LAN1 PXE IPv6 Intel(R) Ethernet Controller I226-LM PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(c87f545af3bf,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot000D* UEFI: LAN2 PXE IPv4 Intel(R) Ethernet Controller I226-LM PciRoot(0x0)/Pci(0x1c,0x1)/Pci(0x0,0x0)/MAC(c87f545af3c0,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000E* UEFI: LAN2 PXE IPv6 Intel(R) Ethernet Controller I226-LM PciRoot(0x0)/Pci(0x1c,0x1)/Pci(0x0,0x0)/MAC(c87f545af3c0,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot000F* UEFI OS HD(2,GPT,232ac1bd-b37a-4bfa-bf2c-0e8b7efbddf1,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
Boot0010* UEFI OS HD(2,GPT,0e7a8476-f6c7-4ede-b992-61101010bcd0,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
root@BKK-PX1:~#

such a noob lol

 

[adresner]

is this section still relevant with 7.4? and systemd?

Step 4: Add Modules​

Many will immediately reboot after the above is done, and it is probably a good practice. Usually, I like to add modules just to save time. If you are more conservative, reboot, then do this step. Next, you will want to add modules by editing:

nano /etc/modules
In that file you will want to add:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd