[SOLVED] IOMMU not enabled on HPE Z640 after following guide

[furmansk]

Hey all,

Have a HP Z640 machine and proxmox 6.4-13 installed. I installed 6.3 at first from ISO but with UEFI option so I assume I am on systemd-boot. I have been trying to get IOMMU enabled and at first I was using the guide and doing all grub commands because I was unaware that I was on systemd-boot instead. But here's where I'm confused.

When I first setup my system with a ZFS raid setup, it wouldn't boot and after searching online I came across adding this line to the grub config file.

GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=10

That fixed my boot issue and it would now find the zfs pool and boot.

Now I've been trying to enable IOMMU and followed the guide, added the extra boot options to that line and enabled it in the bios, nothing worked. Would still get "No IOMMU detected" dialog when adding a PCI device to a VM.

So then I tried editing the systemd-boot cmdline file in /etc/kernel/. When I added the extra parameters, updated the boot with the proxmox-boot-tool command, the system wouldn't boot. Would get stuck with "Reading all physical volumes. This may take awhile...". I even let it sit for a few hours and still no boot.

So my question is, what is it using? Does the grub config file in /etc/default/ get used even with systemd-boot? Why did rootdelay=10 work in the grub and fix my problem?

Once I get this figured out, is there a full up example file for systemd-boot to show how parameters are added? I know it says just add to the end of the line but is that really all?

 

[oguz]

hi,

So my question is, what is it using?

check the output of proxmox-boot-tool status, it will tell you if you're using grub or not.

So then I tried editing the systemd-boot cmdline file in /etc/kernel/. When I added the extra parameters, updated the boot with the proxmox-boot-tool command, the system wouldn't boot. Would get stuck with "Reading all physical volumes. This may take awhile...". I even let it sit for a few hours and still no boot.

which extra parameters did you add? please post the contents from /etc/kernel/cmdline

is there a full up example file for systemd-boot to show how parameters are added? I know it says just add to the end of the line but is that really all?

yes, you just add it to the end of the line. for example my /etc/kernel/cmdline before modifying:

root=ZFS=rpool/ROOT/pve-1 boot=zfs

and after adding iommu parameter:

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

and after changing it you'd run proxmox-boot-tool refresh

 

[furmansk]

Hey oguz,

Thanks for the reply.

Output for proxmox-boot-tool status just states UFEI. Doesn't mention grub or systemd-boot. I assume its systemd-boot based on what the docs state and what my boot up screen looks like.

root@pve:/etc/kernel# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
3939-D6B7 is configured with: uefi (versions: 5.4.128-1-pve, 5.4.78-2-pve)
393A-6FF4 is configured with: uefi (versions: 5.4.128-1-pve, 5.4.78-2-pve)

I started with what the guide said to put in the file.

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

And that's what gave me the "Reading all physical volumes..." stuck booting issue. I will add, I didn't remove anything from the grub file that I had before trying what the guide said to for systemd-boot.

Does it make sense for my system to take the rootdelay=10 in the grub config file but none of the other boot parameters? I even tried adding just the rootdelay to the /etc/kernel/cmdline file at the end of the line to see if that would even work and gave same thing as the first time, "Reading all physical volumes...". For reference here's what my grub config has that I had to setup for rootdelay when I first installed back in January this year.

GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=10 quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction video=efifb:eek:ff"

I added the IOMMU stuff and it hasn't done anything which I assume it won't if I'm on systemd-boot. But still confused as to why rootdelay still works. I guess I could try removing it all even rootdelay and see if it still boots. I'll try that just for my piece of mind.

Thanks.

 

[oguz]

And that's what gave me the "Reading all physical volumes..." stuck booting issue

okay, and does it start working again when you remove the IOMMU option and refresh with boot tool followed by reboot?

if it works without the iommu option but hangs with it enabled, then you might be missing BIOS updates for your motherboard.

also have you enabled "Intel VT-d" option in your BIOS?

 

[furmansk]

okay, and does it start working again when you remove the IOMMU option and refresh with boot tool followed by reboot?

if it works without the iommu option but hangs with it enabled, then you might be missing BIOS updates for your motherboard.

also have you enabled "Intel VT-d" option in your BIOS?

Yeah, so if I remove the IOMMU after booting into a Ubuntu 20.04 live USB and editing it (import zfs, mount partitions etc), it boots back up again. But even if I just add rootdelay=10 to the cmdline file it still hangs at that point.

So I had to use 21.04 Ubuntu to be able to import the rpool since it used latest update/features of ZFS.

Once I did this, I didn't add the rootdelay to the /etc/default/grub file, I instead added it to the /etc/kernel/cmdline file at the end of the line and it booted normally. Still confused as to why if I had it in the /etc/default/grub file it worked too. ¯\_(ツ)_/¯

So now I will try the IOMMU parameters and see. If it doesn't boot then I will look at bios being the problem and try to make sure its updated to the latest.

Thanks.

 

[furmansk]

Updated the BIOS to latest one for this year and still stuck on "Reading all physical volumes...". I thought maybe it was working when the cursor went away thinking that the gpu wasn't loaded like I assume it shouldn't be once booting.

Verified in BIOS that VT-d is enabled.

At a loss on what to do now.

 

[oguz]

Updated the BIOS to latest one for this year and still stuck on "Reading all physical volumes...". I thought maybe it was working when the cursor went away thinking that the gpu wasn't loaded like I assume it shouldn't be once booting.

with updated bios and no IOMMU parameter it still boots normally?

after you've updated BIOS some settings might be reset, make sure you have VMX and VT-d enabled (they might be called differently in your BIOS, best to look it up by model)

which motherboard do you use?

 

[furmansk]

with updated bios and no IOMMU parameter it still boots normally?

after you've updated BIOS some settings might be reset, make sure you have VMX and VT-d enabled (they might be called differently in your BIOS, best to look it up by model)

which motherboard do you use?

Yes still boots fine.

Checked all that and it was set to on as before.

I'm on a HP Z640 workstation with a dual socket motherboard and E5-2678 v3.

And didn't mention this before but trying to pass through a Nvidia 1080 card.

 

[furmansk]

I noticed this yesterday in the kernel logs as I was trying to find out what was happening when it would get stuck booting. Never found that but found this error repeating a bunch after the machine has booted up fine. It is doing it with the GPU and GPU audio device.

Aug  2 13:43:11 pve kernel: [  165.647298] vfio-pci 0000:02:00.0: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Aug  2 13:43:11 pve kernel: [  165.649132] vfio-pci 0000:02:00.0: AER:   device [10de:1b80] error status/mask=00100000/00000000
Aug  2 13:43:11 pve kernel: [  165.650397] vfio-pci 0000:02:00.0: AER:    [20] UnsupReq               (First)
Aug  2 13:43:11 pve kernel: [  165.651670] vfio-pci 0000:02:00.0: AER:   TLP Header: 40000008 000003ff e01c6000 f7f7f7f7
Aug  2 13:43:11 pve kernel: [  165.653009] pcieport 0000:00:02.0: AER: Device recovery successful
Aug  2 13:43:11 pve kernel: [  165.653011] pcieport 0000:00:02.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0

Does this pose a problem with anything related to PCI passthrough?

 

[oguz]

let's start from the beginning.

i assume you're following [0], and you want to passthrough your NVIDIA 1080 to a VM.

please tell me which of the following you did, and what works

1. you've updated BIOS to the latest one for your machine [1] (and this boots normally without issues)

2. you add in your kernel cmdline intel_iommu=on, then do proxmox-boot-tool refresh, and reboot. (this hangs at reading physical volumes?)

can you try the following in your cmdline also and see if it boots: intel_iommu=on iommu=pt video=vesafb:off video=efifb:off

3. you've added the vfio modules in /etc/modules and rebooted, and they show up when you run lsmod | grep vfio

4. nvidia modules are blacklisted in /etc/modprobe.d/pve-blacklist.conf

Does this pose a problem with anything related to PCI passthrough?

that's weird, you could try also setting pci=noaer in the cmdline to see if it makes any difference.

also in the dmesg output do you see anything with IOMMU or DMAR?

5. if you manage to get it booting, you should check find /sys/kernel/iommu_groups/ -type l (described in the wiki, you should see the iommu groups of your devices)

[0]: https://pve.proxmox.com/wiki/Pci_passthrough
[1]: https://support.hp.com/us-en/drivers/selfservice/hp-z640-workstation/6978835[/icode]

 

[furmansk]

i assume you're following [0], and you want to passthrough your NVIDIA 1080 to a VM.

Yes

1. you've updated BIOS to the latest one for your machine [1] (and this boots normally without issues)

Yes

2. you add in your kernel cmdline intel_iommu=on, then do proxmox-boot-tool refresh, and reboot. (this hangs at reading physical volumes?)

Yes

can you try the following in your cmdline also and see if it boots: intel_iommu=on iommu=pt video=vesafb:off video=efifb:off

This worked actually. It rebooted and came back up from web console. No video out to monitor which I assume is because the video=vesafb:off.

3. you've added the vfio modules in /etc/modules and rebooted, and they show up when you run lsmod | grep vfio

Here's the output for this.

root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

root@pve:~# lsmod | grep vfio
vfio_pci               53248  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  19 vfio_pci,kvm
vfio_iommu_type1       32768  0
vfio                   32768  2 vfio_iommu_type1,vfio_pci

4. nvidia modules are blacklisted in /etc/modprobe.d/pve-blacklist.conf

Yes
blacklist nvidiafb
blacklist nouveau

also in the dmesg output do you see anything with IOMMU or DMAR?

Here's the output.

root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.009830] ACPI: DMAR 0x00000000CBFC9000 000140 (v01 HPQOEM SLIC-WKS 00000001 INTL 20091013)
[    0.009856] ACPI: Reserving DMAR table memory at [mem 0xcbfc9000-0xcbfc913f]
[    0.794181] DMAR: Host address width 46
[    0.794183] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.794188] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    0.794189] DMAR: DRHD base: 0x000000f7ffd000 flags: 0x0
[    0.794192] DMAR: dmar1: reg_base_addr f7ffd000 ver 1:0 cap d2008c10ef0466 ecap f0205b
[    0.794193] DMAR: DRHD base: 0x000000f7ffc000 flags: 0x1
[    0.794196] DMAR: dmar2: reg_base_addr f7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    0.794197] DMAR: RMRR base: 0x000000cb02b000 end: 0x000000cb02dfff
[    0.794198] DMAR: ATSR flags: 0x0
[    0.794199] DMAR: ATSR flags: 0x0
[    0.794201] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    0.794202] DMAR-IR: IOAPIC id 8 under DRHD base  0xf7ffc000 IOMMU 2
[    0.794203] DMAR-IR: IOAPIC id 9 under DRHD base  0xf7ffc000 IOMMU 2
[    0.794204] DMAR-IR: HPET id 0 under DRHD base 0xf7ffc000
[    0.794205] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.795013] DMAR-IR: Enabled IRQ remapping in x2apic mode

5. if you manage to get it booting, you should check find /sys/kernel/iommu_groups/ -type l (described in the wiki, you should see the iommu groups of your devices)

Returns nothing after I got it booting with your recommendations. Does this imply no IOMMU is on? I also still see in the GUI that no IOMMU detected.

 

[oguz]

This worked actually. It rebooted and came back up from web console. No video out to monitor which I assume is because the video=vesafb:off.

great, yes the missing video is normal.

also to be sure please run update-initramfs -k all -u to update kernel modules

Returns nothing after I got it booting with your recommendations. Does this imply no IOMMU is on? I also still see in the GUI that no IOMMU detected.

yes you need to see the IOMMU groups, it won't work otherwise. on my machine i see a line like this at the end of the grep:

[    9.453511] DMAR: Intel(R) Virtualization Technology for Directed I/O

are you sure VT-d is enabled in the BIOS? some BIOS will show it as "Auto", but this should be set to "Enabled". also make sure to note the difference between VT-x (virtualization/kvm support) and VT-d (IOMMU support) in your BIOS settings.

 

[furmansk]

Yes, thanks for getting me this far

also to be sure please run update-initramfs -k all -u to update kernel modules

Did that and rebooted. Still no IOMMU groups listed. And when I run the dmesg command I still see what I pasted above.

Yes, I remember it was set to enable. There wasn't an AUTO option. Wonder if I need to reach out to HP and mention there's a bug with this not being enabled correctly.
EDITED: Verified again, yes it was set to Enabled. There was no AUTO option, just ENABLE and DISABLE.

Would having a dual socket machine matter? its a motherboard with a expansion riser card to add another CPU so its dual Xeons.

Thanks for your help.

 

[furmansk]

@oguz , any new thoughts or things to try?

 

[leesteken]

Have you tried this? Maybe you also have a known problem with HP and enabling interrupt remapping and/or IOMMU?
As long as for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done. shows no multiple groups, PCI passthrough will not work because VT-d is not fully enabled.
Can you check with cat /proc/cmdline that all your kernel parameters are actually passed correctly? Just to rule out a /etc/kernel/cmdline versus /etc/default/grub problem.

 

[furmansk]

Have you tried this? Maybe you also have a known problem with HP and enabling interrupt remapping and/or IOMMU?
As long as for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done. shows no multiple groups, PCI passthrough will not work because VT-d is not fully enabled.
Can you check with cat /proc/cmdline that all your kernel parameters are actually passed correctly? Just to rule out a /etc/kernel/cmdline versus /etc/default/grub problem.

Yeah I've tried the link there. Think its a different motherboard. Nothing shows up when I run the command.

On your command, I only get one IOMMU group with a long list of devices.

Yeah I checked /cat/proc/cmdline and its the same as /etc/kernel/cmdline as far as arguments go.

I'm not sure what else to do. I wish I could contact HP and tell them their motherboard is broken and doesn't fully support IOMMU.

 

[leesteken]

IOMMU (VT-d) is clearly not enabled fully for your system (since there are no multiple IOMMU groups).
In earlier posts your showed that you added the necessary kernel parameters to /etc/default/grub. Are the same parameters also in /etc/kernel/cmdline? Did you run update-initramfs -u and reboot? Can you show the output of cat /proc/cmdline just to make sure that this is not the problem? This does happen sometimes.
Maybe you can check the HP support website for information about this or newer BIOS versions? I'm not familiar with HP systems myself, sorry.

 

[furmansk]

IOMMU (VT-d) is clearly not enabled fully for your system (since there are no multiple IOMMU groups).
In earlier posts your showed that you added the necessary kernel parameters to /etc/default/grub. Are the same parameters also in /etc/kernel/cmdline? Did you run update-initramfs -u and reboot? Can you show the output of cat /proc/cmdline just to make sure that this is not the problem? This does happen sometimes.
Maybe you can check the HP support website for information about this or newer BIOS versions? I'm not familiar with HP systems myself, sorry.

Yes they are in my /etc/kernel/cmdline.

Here's the output.

root@pve:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.4.128-1-pve\initrd.img-5.4.128-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs rootdelay=10 quiet iommu=on iommu=pt video=vesafb:off video=efifb:off

Yes I ran update-initramfs -u and rebooted.

Yeah I might reach out over there and see what they say, if anything. Since I have zero support contract with them and this is a home lab setup, I doubt they care. Would be odd though since its a Z640 and not a single person has mentioned it not working with IOMMU. I might just throw in the towel and put it in a physical machine instead. Hate that idea cause only have 2 machines worth putting it in to mine with and one is like a 2009 intel build haha. Other is 2012 AMD build so not much better and its also my media machine currently. Just need to move my data off a windows server and put in a true NAS setup.

Thanks for the help. I still love proxmox! Got my dad using it too. Awesome software!

 

[leesteken]

There is no intel_iommu=on in that line. You need to add that for IOMMU/VFIO/VT-d to work and remove the unnecessary iommu=on (which is incorrect anyway). You can leave the iommu=pt as it sometimes increases performance for non-passedthrough devices.
It might be a good idea to apt install pve-kernel-5.11 (or upgrade to Proxmox VE 7.0) to use the latest supported kernel.

 

[furmansk]

Oh dang lol I thought I had it in there. I'll try that now.

Is there any issues with upgrading to 5.11 kernel? Risks?

Thanks.