Setting up PassThough for SATA/SAS

[stig.j]

Hi Gang.

Im trying to setup a passthough for a sata/sas controller to use for TrueNas Core. but i cannot find a solution, ive searched high and low.

The error im getting:

kvm: -device vfio-pci,host=0000:03:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:03:00.0: failed to setup container for group 51: Failed to set iommu for container: Operation not permitted TASK ERROR: start failed: QEMU exited with code 1


The Computer is a Lenovo ThinkStation P510 with 256GB memory..

This is my currnet config attempt

lspci
03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] (rev 05)

find /sys/kernel/iommu_groups/ -type l | grep 03:00
/sys/kernel/iommu_groups/51/devices/0000:03:00.0

find /sys/kernel/iommu_groups/ -type l | grep /51
/sys/kernel/iommu_groups/51/devices/0000:03:00.0

dmesg | grep 'remapping'
[ 1.008574] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 1.008575] x2apic: IRQ remapping doesn't support X2APIC mode

cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

cat /etc/pve/qemu-server/10504.conf
boot: order=virtio0;ide2;net0
cores: 8
hostpci0: 0000:03:00
ide2: iso:iso/TrueNAS-12.0-U7.iso,media=cdrom
memory: 32768
meta: creation-qemu=6.1.0,ctime=1639323161
name: truenas
net0: virtio=CA:C1:FD:8A:2D:C3,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=c00e1018-347f-41cd-81f5-9da986f025e8
sockets: 1
virtio0: vm4:10504/vm-10504-disk-0.qcow2,size=32G
vmgenid: ee211f80-ced2-4118-af15-0abbd849e12e


cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

 

[stig.j]

No one have any ideas on how I can solve this?

 

[dcsapak]

can you post the complete dmesg, all iommu groups and the output of 'lspci -nnk' ?

 

[stig.j]

Sure, i have attached the files here, as it would have gotten a too big message - chrome was actually struggling while i tried.

 

[dcsapak]

dmesg prints the following:

vfio-pci 0000:03:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

this is most often some bios limitation, so check if there is some switch/option in the bios to allow passthrough etc. ,
check for an updated bios or ask your motherboard vendor

 

[stig.j]

Ok, will do, but PCI passthough worked just fine using ESXi, so i would expect this to work in proxmox as well? - this Workstation was actually running a licensed ESXi 6.5 before.. so any other thoughts?

 

[dcsapak]

older linux kernels (and i guess ESXi) ignored those platform restrictions, but there are technical reasons for it, see this whitepaper from redhat for technical details: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

AFAIK, at least HP has some bios configuration tools where this is configurable, but no idea about Lenovo..