Passing through a LSI SAS Controller

[jackydany]

Hi,

after fixing my W10 OVMF problems with the help of this forum, i am continuing my installtion.

I want to passthrough a LSI HBA SAS Controller 9201-16e to a freenas VM.
I flashed the controller with the latest Firmware, IT Mode. Bios is empty. So its just a HBA in IT Mode passing through as JBOD.
I tested with a hardware Freenas Device, working properly. As devices i am using a Netapp DS4243 with a sbb2-601 controller for the sff-8088 ports.
I inserted a sata drive as well as 2 sas drives and all of them are recognized on the Freenas Hardware.
Also they are recognized in Proxmox properly and i coulid add a disk if i would like to.

But when i try to passing it to the VM, its not working and i cant see why.

cat /etc/pve/qemu-server/105.conf
bios: ovmf
bootdisk: scsi0
cores: 4
efidisk0: wonderland-zfs:vm-105-disk-0,size=1M
hostpci0: 82:00,pcie=1,rombar=0
machine: q35
memory: 16384
name: freetest
net0: virtio=06:0A:D3:D7:61:C6,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsi0: local-zfs:vm-105-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=4f625084-0e02-4102-8789-66af8ebc1ee2
sockets: 2
vmgenid: cd5cc229-4786-41f9-8d4e-bcc8315e172e

lspci:
82:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)

pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

I already passed through a AMD Radeon GPU (2 different ones at the same time) to my new W10 VM. Thats working.
I am passing through 2 USB Sticks to my iobroker VM. Working like hell

I serached a lot but everything i tried didnt succeed.
I also tried romfile=path for the romfile of the card. No success. rombar=1 also. CPU = host same.


Would be nice to get this running. But i dont know where i am stuck

Thanks and happy easter to all of you!

 

[jackydany]

dmesg | grep Adding
[    2.239195] pci 0000:00:00.0: Adding to iommu group 0
[    2.239553] pci 0000:00:01.0: Adding to iommu group 1
[    2.239809] pci 0000:00:02.0: Adding to iommu group 2
[    2.240213] pci 0000:00:02.2: Adding to iommu group 3
[    2.240505] pci 0000:00:03.0: Adding to iommu group 4
[    2.240911] pci 0000:00:03.2: Adding to iommu group 5
[    2.241242] pci 0000:00:05.0: Adding to iommu group 6
[    2.241277] pci 0000:00:05.2: Adding to iommu group 6
[    2.241308] pci 0000:00:05.4: Adding to iommu group 6
[    2.241667] pci 0000:00:11.0: Adding to iommu group 7
[    2.241983] pci 0000:00:1a.0: Adding to iommu group 8
[    2.242351] pci 0000:00:1c.0: Adding to iommu group 9
[    2.242772] pci 0000:00:1c.7: Adding to iommu group 10
[    2.243242] pci 0000:00:1d.0: Adding to iommu group 11
[    2.243516] pci 0000:00:1e.0: Adding to iommu group 12
[    2.245740] pci 0000:00:1f.0: Adding to iommu group 13
[    2.245782] pci 0000:00:1f.2: Adding to iommu group 13
[    2.245814] pci 0000:00:1f.3: Adding to iommu group 13
[    2.246082] pci 0000:01:00.0: Adding to iommu group 14
[    2.246549] pci 0000:03:00.0: Adding to iommu group 15
[    2.246596] pci 0000:03:00.1: Adding to iommu group 15
[    2.246878] pci 0000:04:00.0: Adding to iommu group 16
[    2.247229] pci 0000:04:00.1: Adding to iommu group 17
[    2.247478] pci 0000:06:00.0: Adding to iommu group 18
[    2.247823] pci 0000:06:00.1: Adding to iommu group 19
[    2.248071] pci 0000:06:00.2: Adding to iommu group 20
[    2.248471] pci 0000:06:00.3: Adding to iommu group 21
[    2.248844] pci 0000:07:00.0: Adding to iommu group 22
[    2.249103] pci 0000:08:00.0: Adding to iommu group 23
[    2.249492] pci 0000:08:00.1: Adding to iommu group 24
[    2.249773] pci 0000:0a:00.0: Adding to iommu group 25
[    2.250233] pci 0000:7f:08.0: Adding to iommu group 26
[    2.250281] pci 0000:7f:08.3: Adding to iommu group 26
[    2.250314] pci 0000:7f:08.4: Adding to iommu group 26
[    2.250627] pci 0000:7f:09.0: Adding to iommu group 27
[    2.250663] pci 0000:7f:09.3: Adding to iommu group 27
[    2.250697] pci 0000:7f:09.4: Adding to iommu group 27
[    2.251136] pci 0000:7f:0a.0: Adding to iommu group 28
[    2.251172] pci 0000:7f:0a.1: Adding to iommu group 28
[    2.251205] pci 0000:7f:0a.2: Adding to iommu group 28
[    2.251238] pci 0000:7f:0a.3: Adding to iommu group 28
[    2.251540] pci 0000:7f:0b.0: Adding to iommu group 29
[    2.251576] pci 0000:7f:0b.3: Adding to iommu group 29
[    2.252085] pci 0000:7f:0c.0: Adding to iommu group 30
[    2.252121] pci 0000:7f:0c.1: Adding to iommu group 30
[    2.252155] pci 0000:7f:0c.2: Adding to iommu group 30
[    2.252189] pci 0000:7f:0c.6: Adding to iommu group 30
[    2.252224] pci 0000:7f:0c.7: Adding to iommu group 30
[    2.252621] pci 0000:7f:0d.0: Adding to iommu group 31
[    2.252659] pci 0000:7f:0d.1: Adding to iommu group 31
[    2.252695] pci 0000:7f:0d.2: Adding to iommu group 31
[    2.252730] pci 0000:7f:0d.6: Adding to iommu group 31
[    2.253008] pci 0000:7f:0e.0: Adding to iommu group 32
[    2.253044] pci 0000:7f:0e.1: Adding to iommu group 32
[    2.253532] pci 0000:7f:0f.0: Adding to iommu group 33
[    2.253571] pci 0000:7f:0f.1: Adding to iommu group 33
[    2.253609] pci 0000:7f:0f.2: Adding to iommu group 33
[    2.253646] pci 0000:7f:0f.3: Adding to iommu group 33
[    2.253684] pci 0000:7f:0f.4: Adding to iommu group 33
[    2.253721] pci 0000:7f:0f.5: Adding to iommu group 33
[    2.253758] pci 0000:7f:0f.6: Adding to iommu group 33
[    2.254192] pci 0000:7f:10.0: Adding to iommu group 34
[    2.254232] pci 0000:7f:10.1: Adding to iommu group 34
[    2.254277] pci 0000:7f:10.2: Adding to iommu group 34
[    2.254316] pci 0000:7f:10.3: Adding to iommu group 34
[    2.254354] pci 0000:7f:10.4: Adding to iommu group 34
[    2.254393] pci 0000:7f:10.5: Adding to iommu group 34
[    2.254432] pci 0000:7f:10.6: Adding to iommu group 34
[    2.254472] pci 0000:7f:10.7: Adding to iommu group 34
[    2.254865] pci 0000:7f:11.0: Adding to iommu group 35
[    2.255231] pci 0000:7f:13.0: Adding to iommu group 36
[    2.255270] pci 0000:7f:13.1: Adding to iommu group 36
[    2.255309] pci 0000:7f:13.4: Adding to iommu group 36
[    2.255350] pci 0000:7f:13.5: Adding to iommu group 36
[    2.255392] pci 0000:7f:13.6: Adding to iommu group 36
[    2.255780] pci 0000:80:00.0: Adding to iommu group 37
[    2.256023] pci 0000:80:01.0: Adding to iommu group 38
[    2.256378] pci 0000:80:01.1: Adding to iommu group 39
[    2.256623] pci 0000:80:02.0: Adding to iommu group 40
[    2.256984] pci 0000:80:03.0: Adding to iommu group 41
[    2.257303] pci 0000:80:05.0: Adding to iommu group 42
[    2.257349] pci 0000:80:05.2: Adding to iommu group 42
[    2.257390] pci 0000:80:05.4: Adding to iommu group 42
[    2.257731] pci 0000:82:00.0: Adding to iommu group 43
[    2.258167] pci 0000:84:00.0: Adding to iommu group 44
[    2.258220] pci 0000:84:00.1: Adding to iommu group 44
[    2.258528] pci 0000:85:00.0: Adding to iommu group 45
[    2.258580] pci 0000:85:00.1: Adding to iommu group 45
[    2.258997] pci 0000:ff:08.0: Adding to iommu group 46
[    2.259043] pci 0000:ff:08.3: Adding to iommu group 46
[    2.259088] pci 0000:ff:08.4: Adding to iommu group 46
[    2.259402] pci 0000:ff:09.0: Adding to iommu group 47
[    2.259448] pci 0000:ff:09.3: Adding to iommu group 47
[    2.259495] pci 0000:ff:09.4: Adding to iommu group 47
[    2.259950] pci 0000:ff:0a.0: Adding to iommu group 48
[    2.259996] pci 0000:ff:0a.1: Adding to iommu group 48
[    2.260044] pci 0000:ff:0a.2: Adding to iommu group 48
[    2.260088] pci 0000:ff:0a.3: Adding to iommu group 48
[    2.260379] pci 0000:ff:0b.0: Adding to iommu group 49
[    2.260424] pci 0000:ff:0b.3: Adding to iommu group 49
[    2.260869] pci 0000:ff:0c.0: Adding to iommu group 50
[    2.260915] pci 0000:ff:0c.1: Adding to iommu group 50
[    2.260960] pci 0000:ff:0c.2: Adding to iommu group 50
[    2.261005] pci 0000:ff:0c.6: Adding to iommu group 50
[    2.261051] pci 0000:ff:0c.7: Adding to iommu group 50
[    2.261395] pci 0000:ff:0d.0: Adding to iommu group 51
[    2.261442] pci 0000:ff:0d.1: Adding to iommu group 51
[    2.261492] pci 0000:ff:0d.2: Adding to iommu group 51
[    2.261538] pci 0000:ff:0d.6: Adding to iommu group 51
[    2.261954] pci 0000:ff:0e.0: Adding to iommu group 52
[    2.262003] pci 0000:ff:0e.1: Adding to iommu group 52
[    2.262522] pci 0000:ff:0f.0: Adding to iommu group 53
[    2.262572] pci 0000:ff:0f.1: Adding to iommu group 53
[    2.262619] pci 0000:ff:0f.2: Adding to iommu group 53
[    2.262665] pci 0000:ff:0f.3: Adding to iommu group 53
[    2.262713] pci 0000:ff:0f.4: Adding to iommu group 53
[    2.262761] pci 0000:ff:0f.5: Adding to iommu group 53
[    2.262810] pci 0000:ff:0f.6: Adding to iommu group 53
[    2.263255] pci 0000:ff:10.0: Adding to iommu group 54
[    2.263308] pci 0000:ff:10.1: Adding to iommu group 54
[    2.263356] pci 0000:ff:10.2: Adding to iommu group 54
[    2.263405] pci 0000:ff:10.3: Adding to iommu group 54
[    2.263455] pci 0000:ff:10.4: Adding to iommu group 54
[    2.263506] pci 0000:ff:10.5: Adding to iommu group 54
[    2.263555] pci 0000:ff:10.6: Adding to iommu group 54
[    2.263604] pci 0000:ff:10.7: Adding to iommu group 54
[    2.264002] pci 0000:ff:11.0: Adding to iommu group 55
[    2.264379] pci 0000:ff:13.0: Adding to iommu group 56
[    2.264430] pci 0000:ff:13.1: Adding to iommu group 56
[    2.264481] pci 0000:ff:13.4: Adding to iommu group 56
[    2.264530] pci 0000:ff:13.5: Adding to iommu group 56
[    2.264578] pci 0000:ff:13.6: Adding to iommu group 56
[   16.566400] ipmi_si: Adding SMBIOS-specified kcs state machine
[   16.566941] ipmi_si: Adding ACPI-specified kcs state machine

 

[tonyrich]

Did you ever figure this out? I'm looking to do the same thing, without passing individual disks. Guess I'll go back to google.

EDIT: Found this and it worked great! https://pve.proxmox.com/wiki/Pci_passthrough
Had to disable boot options on the HBA before VirtIO SCSI would load the os.
That's it!

 

[Huamijo]

i don't know if this was the correct solution, but for me it was resolved by unchecking "ROM-Bar" hidden under "advanced".

Maybe this wasn't the right solution, but for now it works and after 5 min their won't be any error about "unable to load MPT BIOS".

 

[Blackice504]

As the rom is loaded on the host machine you do not need the rom on the guest and might cause issues such as not working, I did the same i did the pass-through and untick the rom, when the rom was ticked it did not work, also make sure you have under hardware Q35,viommu= your host cpu, there is virtio selection but i have never tried it i am guessing that is for processors that do not support hardware viommu functions, i could be wrong.

But as stated what worked for me untick rom under advanced tab of pci device so it should look like HOSTPCI0 000:01:00,rombar=0

and should work, also do this for the best performance as passthrough individual disks or having large raw files for virtual images on the disks adds overhead that is not needed, especially with this configuration.

 

[BuBs626]

I disabled the "ROM-Bar" setting under "advanced" when adding PCI device for the VM. My TrueNAS VM is no longer throwing errors and seems to be stable after a few hours. I am passing a 9400-16i.