Failed to start zfs-import..port ZFS pools by device scanning after setting IOMMU=pt

[jaytee129]

I installed Proxmox 8 on a new AMD Ryzen 5 5500GT based machine with ASRock B550M Pro SE mb using UEFI boot with two 500GB drives in ZFS RAID 1.

Things generally work until I try to get PCI Passthrough working by adding "quiet iommu=pt" to /etc/kernel/cmdline, per this and other tutorials

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet iommu=pt

Then I get the following error at boot time. It doesn't prevent the ultimate booting but does delay it and does suggest something I could need one day isn't going to work at some point, like maybe booting off mirror if main drive fails? Maybe something worse?



Yes, I have IOMMU and AMD-V/AMD SVM enabled in BIOS. Also, FWIW, this is after updating OS to latest version available to me as of right now (Linux 6.8.12-1-pve (2024-08-05T16:17Z)).

For a couple weeks I had the machine running with no errors with PCI Passthrough using Legacy boot and GRUB. I re-installed to change to UEFI/systemd-boot after reading that use of GRUB is quite sub optimal with ZFS RAID 1 boot disks.

That said,I wasn't able to get separate isolated IOMMU groups for each device, as I read one is supposed to get with full proper PCI Passthrough. Despite this, though, the passed through LSI Host Bus Adapter (connecting SSDs for my testing) operated much faster when I had the machine configured with Legacy/GRUB/IOMMU=pt. The performance of the HBA is significantly lower now without PCI Passthrough enabled.

For what it's worth, my main intended use for this machine is for a TrueNAS VM. I will also run some some utilities, e.g. Unifi Controller and other services, that I was running in Jails on a TrueNAS box and elsewhere. The only thing I'm looking to Passthrough (at least right now) is the HBA.

Aside from the performance issue, I worry the fact the HBA is now shareable with other VMs to be a recipe for problems. The device should really only be usable/used by the TrueNAS VM.

What does the error mean, practically speaking?
Why does enabling IOMMU cause this?
What could be done to fix it?

Any info would be appreciated.

 

[jaytee129]

Is there a solution for this?

if not, is there an explanation and/or insights that might help me figure out a solution?

 

[leesteken]

iommu=pt does not really do much (if at all) on most systems. It would really surprise me if it caused your problem. Maybe just remove it again to make sure?

 

[jaytee129]

iommu=pt does not really do much (if at all) on most systems. It would really surprise me if it caused your problem. Maybe just remove it again to make sure?

interesting. I've been doing some testing without it (while I've been waiting for responses) and my disk performance via the HBA is about 60% of what it was when I did have iommu=pt.

I don't imagine performance issues are because of a systemd-boot vs grub thing, nor a EFI vs Legacy boot - is it?

Also, why does it say to do that in the online documentation for PCI Passthrough? for example here, plus many tutorials, user posts, etc.

Is this new for version 8?

 

[leesteken]

interesting. I've been doing some testing without it (while I've been waiting for responses) and my disk performance via the HBA is about 60% of what it was when I did have iommu=pt.

It sets up an identity mapping for non-passed through devices (instead of what it does otherwise) and you are the first person I know that reported a (major) performance difference!

I don't imagine performance issues are because of a systemd-boot vs grub thing, nor a EFI vs Legacy boot - is it?

Since the device firmware can differ between BIOS en UEFI, that could also have an effect on performance.

Also, why does it say to do that in the online documentation for PCI Passthrough? for example here, plus many tutorials, user posts, etc.

People seem to think it means "set IOMMU to PassThrough" and everybody copies other people. I believe it was worded differently (and incorrectly or at least confusingly, IMHO) in older Proxmox manuals. When I asked people why they used it, the answer was always "other people told them" and "it does no harms" and "maybe it helps with passthrough" (which is hilarious since it only works for devices that are not passed through).

Is this new for version 8?

No, this has been a optional Linux kernel parameter for a long time. But it first appeared in the Proxmox VE 7 manual.

 

[jaytee129]

So is adding the following to /etc/modules - also in the documentation - wrong/unnecessary too?

vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd #not needed if on kernel 6.2 or newer

I will say that I did play with some BIOS options before the latest proxmox rebuild in the hope it would enable ACS (also mentioned in documentation) because it's not an explicitly labeled thing. Maybe one of those messed with the HBA performance. That or maybe I need to re-install TrueNAS (what I'm testing with) *after* I've made all the needed lower level changes. I'll try all that.

Unfortunately, though, I'm now completely confused as to what I need to do in proxmox to enable PCI Passthrough. Particularly since I did all the things the documentation/other people said to enable PCI Passthrough on my SuperMicro / AMD EPYC based proxmox 7 box and it has been working beautifully for well over a year now. (2 Video cards and a couple of USB devices have been passthrough to different VMs successfully)

Where can I get definitive information on enabling PCI Passthrough in Proxmox 8?

 

[jaytee129]

going back to my original post, I went back and reenable TPM in BIOS and re-installed px, which caused installation to be with GRUB this time and I'm no longer getting the error at boot up, either with or without iommu=pt.

So I guess that is the fix for this particular issue but I'm not sure what it means in terms of appropriate support for zfs. In a number of posts, including from one or more "Distinguished Members" it's been said that zfs and grub were not a great pairing. I guess without a fix for this issue with systemd-boot I don't have much of a choice.

 

[alexskysilk]

it's been said that zfs and grub were not a great pairing.

grub needs to be installed on a non zfs partition. UEFI is more flexible.

going back to my original post, I went back and reenable TPM in BIOS

I didnt see any mention of tpm in your initial post. since BIOS more has no provision wrt to TPM, it would make sense that your host would boot in bios mode regardless of TPM.

Long story short- you need to be aware of what you're doing and the consequences thereof. if you dont share that information when you ask for help, its not like anyone can guess.

 

[jaytee129]

I didnt see any mention of tpm in your initial post. since BIOS more has no provision wrt to TPM, it would make sense that your host would boot in bios mode regardless of TPM.

Long story short- you need to be aware of what you're doing and the consequences thereof. if you dont share that information when you ask for help, its not like anyone can guess.

Didn't think TPM was relevant to the issue. I believe it got disabled when I disabled Secure Boot, which is required for the install to use systemd-boot. Otherwise proxmox install sets things up with GRUB. Sorry you didn't know that. i did my best to provide as much detail/context as I could without going on and on.

I should also stress that the error did not happen until i set iommu=pt. In fact proxmox rebooted several times without the error before that. To me the most relevant points are zfs mirror and iommu=pt. The rest was more context to head off the "what are you trying to do?" and "did you turn virtualization on in the BIOS?" questions. If something else was important and relevant I would have expected someone to ask about it rather than scold/lecture on not proactively mentioning it.

 

[jaytee129]

grub needs to be installed on a non zfs partition. UEFI is more flexible.

to be clear about my end, booting is still using UEFI (not Legacy). doesn't proxmox create what it needs?

 

[leesteken]

So is adding the following to /etc/modules - also in the documentation - wrong/unnecessary too?

vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd #not needed if on kernel 6.2 or newer

The last one does not exist anymore and there have been reports that he other three are loaded automatically when a VM with passthrough starts.

Unfortunately, though, I'm now completely confused as to what I need to do in proxmox to enable PCI Passthrough. Particularly since I did all the things the documentation/other people said to enable PCI Passthrough on my SuperMicro / AMD EPYC based proxmox 7 box and it has been working beautifully for well over a year now. (2 Video cards and a couple of USB devices have been passthrough to different VMs successfully)

There is not much difference between Proxmox versions as it is a Linux feature (but Proxmox makes it easier to use). Since kernel 6.8 intel_iommu=on is also no longer needed since it is on by default. With a modern AMD CPU and devices that reset properly and work well with passthrough sometimes no changes are necessary.

Where can I get definitive information on enabling PCI Passthrough in Proxmox 8?

It's very much hardware specific (since you are connecting physical devices to your VMs) and many devices need work-arounds to work with passthrough.
I don't think there is a final guide for passthrough and I would normally suggest taking a minimalistic approach, but your iommu=pt performance issue really surprised me.

 

[alexskysilk]

to be clear about my end, booting is still using UEFI (not Legacy). doesn't proxmox create what it needs?

It does. the addition of tpm into the mix means updates to your bootloader can only be made inside the booted system or else they will not be allowed to boot. for home use, you probably want to just avoid it ever being on.

I should also stress that the error did not happen until i set iommu=pt. In fact proxmox rebooted several times without the error before that. To me the most relevant points are zfs mirror and iommu=pt.

herein lies the issue; you're asking for help, but already decided the solution and omitted data that could have been relevent. I dont know if the addition of that kernel line argument had anything to do with your issue or not, and I have no way of telling- but I gather this is now a moot point since your system boots normally.