[SOLVED] Frustrated. LSI SAS passthrough is not working

[ShinigamiLY]

Hallo there. I bought a new server (DL380p G8) and different sas controller. One of them I need to passthrough but I get always the same error and I rly need help cuz I`m crying

kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=pci.0,addr=0x10,rombar=0: vfio 0000:07:00.0: failed to setup container for group 32: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1

System informations​

System: HP ProLiant DL380 G8
OS: ProxMox 7.0-1 (atm) & 6.4-1 (tested)
Attached devices: Nothing except the LSI SAS Controller to get passthrough. For testing purposes proxmox is installed on an usb stick.

I am able to passthrough like the USB but not the SCSI Controller. On the controller any disks are not atm attached. Even so it isnt working.

Lets get started with what I already did...
Guidelines: Pci passthrough - proxmox Guide ; PCI Passthrough Selection with Identical Devices ; PCI(e) Passthrough; PCI passthrough via OVMF ;

nano /etc/default/grub →

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=1000:0087 vfio-pci.ids=1000:0087 vfio_iommu_type1.allow_unsafe_interrupts=1 nofb video=vesafb:off,efifb:off"

nano /etc/modules →

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
echo "options vfio-pci ids=1000:0087" > /etc/modprobe.d/vfio.conf
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

update-grub
update-initramfs -u -k all
pve-efiboot-tool refresh

reboot

Output informations​

DELETED

If more infos needed, just ask but pls help me out

 

[dcsapak]

[ 0.000000] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
[ 0.000000] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff]

seems your bios is not in an optimal state for passthrough... sadly some vendors do often ship bios versions where such things do not work correcltly
i'd check if there is a newer bios available for your mainboard
as well as checking all iommu/pci related settings in your bios (e.g. above 4g decoding, etc..)

 

[ShinigamiLY]

seems your bios is not in an optimal state for passthrough... sadly some vendors do often ship bios versions where such things do not work correcltly
i'd check if there is a newer bios available for your mainboard
as well as checking all iommu/pci related settings in your bios (e.g. above 4g decoding, etc..)

I already checked the bios version.
I have the latest
871795_001_spp-2017.04.0-SPP2017040.2017_0420.14
and
P03093_001_spp-Gen8.1-SPPGen81.4
also in the bios settings. I checked multiple times everything what has to do with virtualisation (maybe I overlooked someone in this strange bios).
that would be rly sad if this wouldnt work...
I even tried multiple / different PCI HBA Controller for this. None worked.

Only as I said. Stuf like USB it works to passthrough.

Edit: After u told me about this bios / rmrr problem, I googled a bit about this and found an long thread... Need to check this one then. If someone has other "faster help" u`r wellcome.

Compile Proxmox VE with patched intel-iommu driver to remove RMRR check

 

[ShinigamiLY]

I finally did it...
The part with a patched kernel worked for me too. I created my own one but the easier way is to follow the link from:
https://github.com/kiler129/relax-intel-rmrr/blob/master/README.md#installation

This could maybe be something it could be nice already patched in proxmox kernel. Seams like esxi is doing the same.

 

[ShinigamiLY]

My "easy" roadtrip guide

ssh root@ip

mkdir /usr/src/custom-kernel

cd /usr/src/custom-kernel

Get latest version → https://github.com/kiler129/relax-intel-rmrr/releases/

// current ones but need to be edited by the wanted kernel version

wget https://github.com/kiler129/relax-intel-rmrr/releases/download/v1.0.6/linux-tools-5.4-dbgsym_5.4.124-1_amd64.deb
wget https://github.com/kiler129/relax-intel-rmrr/releases/download/v1.0.6/linux-tools-5.4_5.4.124-1_amd64.deb
wget https://github.com/kiler129/relax-intel-rmrr/releases/download/v1.0.6/pve-headers-5.4.124-1-pve-relaxablermrr_5.4.124-1_amd64.deb
wget https://github.com/kiler129/relax-intel-rmrr/releases/download/v1.0.6/pve-kernel-5.4.124-1-pve-relaxablermrr_5.4.124-1_amd64.deb
wget https://github.com/kiler129/relax-intel-rmrr/releases/download/v1.0.6/pve-kernel-libc-dev_5.4.124-1_amd64.deb

dpkg -i *.deb

nano /etc/default/grub →

GRUB_DEFAULT=saved
GRUB_TIMEOUT=3
GRUB_SAVEDEFAULT=true
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intremap=no_x2apic_optout nofb video=vesafb:off,efifb:off"
# Comment this out
#GRUB_DISABLE_RECOVERY="true"

nano /etc/modules →

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

lspci | grep -e SAS

This way it works even with multiple identical devices

find /sys/devices/pci* | grep 0a:00.0 → pci0000:00/0000:00:1c.0/0000:0a:00.0

nano /etc/initramfs-tools/scripts/init-top/bind_vfio.sh

#!/bin/sh -e
echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:1c.0/0000:0a:00.0/driver_override

modprobe -i vfio-pci

chmod 755 /etc/initramfs-tools/scripts/init-top/bind_vfio.sh
chown root:root /etc/initramfs-tools/scripts/init-top/bind_vfio.sh

nano /etc/initramfs-tools/modules

vfio-pci

Normally not all 3 needed but I like to be sure...

update-grub
update-initramfs -u -k all
proxmox-boot-tool refresh

proxmox-boot-tool kernel list

reboot

Grub Menu → Advanced

Select the “new” Kernel from the list. This should now be saved for the next boot and so on. Thats said the last selected one gona be always the default one for the next boots.

dmesg | grep -e DMAR -e IOMMU

→ DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss

lspci -s 0a:00.0 -v

Kernel driver in use: vfio-pci

For a self created Kernel →
https://forum.proxmox.com/threads/compile-proxmox-ve-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/