Enabling IOMMU for PCI_e Device Pass-though.

[Windows7ge]

Software: PROXMOX 6.0-4
Hardware: 2xIntel Xeon E5-2670v1, ASRock Rack EP2C602-4L/D16

In the BIOS I have enabled VT-d. This brings up ATS Support & Coherency Support. I've looked into these and I don't think they're necessary. but please correct me there.

I opened up the Proxmox documentation and followed it as accurately as I could and yet the WebUI still reports:

No IOMMU detected, please activate it. See Documentation for further information.

If I try to attach a PCI_e device anyways:

Error: IOMMU not present

1. I added "intel_iommu=on" to GRUB_CMDLINE_LINUX_DEFAULT="quiet" in /etc/default/grub (was I suppose to use systemd-boot?)
2. I added vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd to /etc/modules
3. ran "update-initramfs -u -k all" (& "pve-efiboot-tool refresh")
4. rebooted
5. ran "dmesg | grep -e DMAR -e IOMMU -e AMD-Vi" which brought up this:

6. [    0.033682] ACPI: DMAR 0x000000007DF949E8 000108 (v01 A M I  OEMDMAR  00000001 INTL 00000001)
   [    1.450866] DMAR: Host address width 46
   [    1.450867] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
   [    1.450873] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fa
   [    1.450874] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
   [    1.450877] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0462 ecap f020fa
   [    1.450878] DMAR: RMRR base: 0x0000007f231000 end: 0x0000007f23efff
   [    1.450879] DMAR: RHSA base: 0x000000fbffe000 proximity domain: 0x1
   [    1.450880] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
   [    1.450883] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffe000 IOMMU 0
   [    1.450884] DMAR-IR: IOAPIC id 0 under DRHD base  0xdfffc000 IOMMU 1
   [    1.450885] DMAR-IR: IOAPIC id 2 under DRHD base  0xdfffc000 IOMMU 1
   [    1.450886] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
   [    1.450887] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
   [    1.451457] DMAR-IR: Enabled IRQ remapping in x2apic mode

7. Checked with "find /sys/kernel/iommu_groups/ -type l" (brought up nothing)
8. Skipped Unsafe interrupts.
9. blacklisted the GPU drivers system wide
10. updated initramfs
11. rebooted

And nothing...still reports that it's not enabled. I'm at a loss. I don't know where to go from here.

 

[n1nj4888]

Are you booting the host using UEFI or legacy bios boot? If using UEFI, you need to add the kernel command line parameters in

/etc/kernel/cmdline

rather than

/etc/default/grub

. You then need to run

pve-efiboot-tool refresh

and reboot.

The following wiki should be useful (note the link under IOMMU section which mentions the different options for changing the kernel command line parameters...

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

 

[Windows7ge]

@n1nj4888 It's the C602 chipset which i believe boarders when UEFI first started coming out so I don't know if it's booting using UEFI or not but it's worth trying.

What you've linked is what I've already read in the documentation.

Checking the /etc/kernel/ directory cmdline does not exist as a file. Is this something I need to create? Would you happen to know what lines I need to write (inside cmdline)? The documentation does not outline this as far as I'm able to see.

 

[n1nj4888]

Normally you set whether it’s UEFI/bios boot in the bios itself. If the directory /sys/firmware/efi exists and is populated then you’re using UEFI boot.

If you’re using UEFI then you need to update the contents of the Kernel command line - the contents of my /etc/kernel/cmdline file are as follows but given you don’t have a /etc/kernel/cmdline file already I suspect you are booting using legacy bios. If you are using UEFI, the example file contents would need to be updated to match your root drive format/setup.

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

 

[mbosma]

I have the exact same setup as you have, mine works fine with pci-e passthrough.
There's something you should check, 1st gen 2670 cpu's have a known bug wich breaks VT-d:
SR0H8 is C1 stepping, VT-d is broken in this stepping. SR0KX is the C2 stepping where VT-d is fixed.

My server isn't on at this moment but I'll try to give you the output of mine when I find the time to spin it up.

 

[Windows7ge]

If the directory /sys/firmware/efi exists and is populated then you’re using UEFI boot.

It does not. It does appear I'm using a UEFI BIOS (looks like the classic BIOS though) but it's not showing me a UEFI boot option for the SATA controller I'm using.

Going into the Boot tab I see "CSM parameters" which I've set to UEFI only but I don't know if that will install the OS under UEFI. If @mbosma has my motherboard maybe he knows exactly how to enable UEFI boot.

If I switch my SATA controller for the built-in Marvel it does have a UEFI option so I wonder if I have no choice but to use that. I used dd to create the bootable media. Is there a more specific way to set the bootable media as UEFI not Legacy? Last I found Rufus does not work with PROXMOX but it does have specifically a UEFI boot option.

I have the exact same setup as you have, mine works fine with pci-e passthrough.
There's something you should check, 1st gen 2670 cpu's have a known bug wich breaks VT-d:
SR0H8 is C1 stepping, VT-d is broken in this stepping. SR0KX is the C2 stepping where VT-d is fixed.

While researching my problem I did come across this. I do believe the stepping is engraved in the CPU's IHS but I'd really rather not have to pop the coolers off to check. Running cpuinfo it says I have a Stepping of 7...I really hope that isn't Hex otherwise I'm well below C2

 

[mbosma]

I am running bios because uefi on pve 5 gave me some issues with booting.
Here's a snippet of my bash history wich results in a working config:

vim /etc/default/zfs
vim /etc/default/grub
update-grub
vim /etc/modules
dmesg | grep ecap
find /sys/kernel/iommu_groups/ -type l
ls /sys/kernel/iommu_groups/
vim /etc/default/grub
lspci
lspci -n -s 81:00
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "options vfio-pci ids=10de:1401,10de:0fba disable_vga=1" > /etc/modprobe.d/vfio.conf
echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf
dmesg
echo vfio >> /etc/modules
echo vfio_iommu_type1 >> /etc/modules
echo vfio_pci >> /etc/modules
echo vfio_virqfd >> /etc/modules
update-initramfs -u
update-grub
reboot

 

[Windows7ge]

I am running bios because uefi on pve 5 gave me some issues with booting.
Here's a snippet of my bash history wich results in a working config:

...the documentation lied to me. It said running "pve-eftboot-tool refresh" was the same as running update-grub. Well the two different update commands I used apparently didn't do this because I decided to run update-grub and all of a sudden everything is working. Passing-though the GPU took a little bit of setting & driver tinkering but it's in, driver's installed, and I got PARSEC going and I'm able to connect to the VM over the network.

 

[mbosma]

Nice!
What gpu are you using?
My gtx 960 has been giving me some annoying code 43 errors while my quadro p2000 works like a charm.

 

[Windows7ge]

Nice!
What gpu are you using?
My gtx 960 has been giving me some annoying code 43 errors while my quadro p2000 works like a charm.

Just a really old Radeon HD 7970 because it's what I have on hand/available. I plan to buy a half-height RX 560. This VM doesn't need high GPU compute performance just enough to run some basic 3D/OpenCL applications.

While exploring GPU pass-though you never learned about the infamous NVIDIA error code 43? I don't have all the details but from what I've read NVIDIA in their infinite wisdom decided to deliberately limit their hardware's use cases by making it so all of their desktop variants will detect weather or not it is in a VM. If it does it will refuse to work and report an error with code 43.

Supposedly there are ways to trick the GPU by making it think it's not in a VM but I've been told success with this is hit or miss.

 

[mbosma]

Supposedly there are ways to trick the GPU by making it think it's not in a VM but I've been told success with this is hit or miss.

Exactly, in my case it has been a miss.
I've had great luck with quadro cards but gtx cards not so much.

 

[Boby_Ross]

I just wanted to add my experience with setting this up. I'm using proxmox 6.1-5 and did the same steps mentioned in the guides (here, and here), but couldn't get it to enable iommu. In the end i was able to get it to work by doing

update-grub2

instead of update-grub like in the manual. I hope this helps others.

 

[Stoiko Ivanov]

instead of update-grub like in the manual. I hope this helps others.

both should be identical on a clean install:

diff -s /usr/sbin/update-grub /usr/sbin/update-grub2 
Files /usr/sbin/update-grub and /usr/sbin/update-grub2 are identical

 

[Boby_Ross]

both should be identical on a clean install

I did the check and you are right! So i checked what i did last night, and it turned out that instead of update-grub i actually did
grub-mkconfig which didn't work.
Thanks for pointing that out.