PCIe Passtrough on ZFS not working

[Murphi]

Hello,

I am having troubles with PCIe Passtrough. I followed your guides on this Topic without any luck.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
https://pve.proxmox.com/wiki/Pci_passthrough
https://pve.proxmox.com/wiki/Host_Bootloader

My Setup:
2x Xeon E5-2680
ASRock EP2C602-4L/D16
Proxmox VE 6.x in UEFI Mode on an ZFS.
VT-d is enabled

According to your guide, I am supposed to add "Intel_iommu=on" in '/etc/kernel/cmdline' like so "root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on ". And run 'pve-efiboot-tool refresh' after that. After editing '/etc/modules' and runing update-initramfs -u -k all" I reboot my system.
However, If I do that my Server will only boot into 'initramfs' and "zpool list" tells me "no pools available".


I tried it with grub too, but that didn't work either.

"cat /proc/cmdline" showswithout any editing from me, or when using grub)

initrd=\EFI\proxmox\5.3.10-1-pve\initrd.img-5.3.10-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

'dmesg' pastebin: https://pastebin.com/zeaVHf4x

'ls -l /sys/kernel/iommu_groups' returns nothing


'zfs list | grep pve-1' returns:

rpool/ROOT/pve-1 1013M 107G 1013M /

I just tried it without ZFS. But I am also having the same boot Problem when using grub. I only get into initramfs.

I would appreciate some help, I am out of ideas. Thank you in advance.

 

[MasterPhi]

Maybe try add BOOT_IMAGE

BOOT_IMAGE=/ROOT/pve-1/@//boot/pve/vmlinuz ro ramdisk_size=16777216 root=ZFS=rpool/ROOT/pve-1 boot=zfs nmi_watchdog=0 amd_iommu=on iommu=pt

 

[Murphi]

Can I paste this or do I have to change something?

BOOT_IMAGE=/ROOT/pve-1/@//boot/pve/vmlinuz ro ramdisk_size=16777216 root=ZFS=rpool/ROOT/pve-1 boot=zfs nmi_watchdog=0 amd_iommu=on iommu=pt

 

[MasterPhi]

Can I paste this or do I have to change something?

BOOT_IMAGE=/ROOT/pve-1/@//boot/pve/vmlinuz ro ramdisk_size=16777216 root=ZFS=rpool/ROOT/pve-1 boot=zfs nmi_watchdog=0 amd_iommu=on iommu=pt

This is my config, at least you need to use intel_iommu instead of amd_iommu

 

[Murphi]

Unfortunately it didn`t work. Would it be possible that you look at the video?

https://www.youtube.com/watch?v=65ahQtBrfMY&feature=youtu.be


I tried some more things, If I write amd_iommu instead of intel_iommu, PCIe Passtrough doesnt work obviously. But the Server boots. So I think I must have some wrong BIOS Settings. Do you happen to know yours?

 

[MasterPhi]

What's your disk layout ? Using any raid controller, HBA ?

 

[Murphi]

What's your disk layout ? Using any raid controller, HBA ?

only 2x SSD in a ZFS and 2x HDD in a ZFS. No Raid Controler

 

[MasterPhi]

I had a X9SRL-F (also C602) and I haven't any issues... Have you done anything else in the guide, for example blacklist anything ?

 

[Murphi]

Well, not yet. It won`t boot if I modify /etc/kernel/cmdline

 

[MasterPhi]

I'm out if idea......

 

[MasterPhi]

Seems like something to do with this.
https://bugzilla.kernel.org/show_bug.cgi?id=42679
And

it sounds like the 9230 has other problems in addition to this one

 

[Murphi]

Seems like something to do with this.
https://bugzilla.kernel.org/show_bug.cgi?id=42679
And

That did it thank you.