Pass-through of 2x PCIe LSI controllers failing

Jan 22, 2017
2
0
21
36
I'm trying to get these both to pass-through to a VM:

Code:
root@pve:~# lspci | grep LSI
26:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
27:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
root@pve:~#

Proxmox is installed on a 32GB USB flash drive.
Added the Proxmox repo. Repo contents not included because my account is too new.


Did a full upgrade. All packages and the kernel.

Code:
root@pve:~# pveversion -v
proxmox-ve: 4.4-82 (running kernel: 4.4.40-1-pve)
pve-manager: 4.4-12 (running version: 4.4-12/e71b7a74)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.40-1-pve: 4.4.40-82
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-109
pve-firmware: 1.1-10
libpve-common-perl: 4.0-92
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-94
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-3
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
root@pve:~#
END


Updated GRUB with:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Then ran 'update-grub'

Added the modules:
Code:
root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.


vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
root@pve:~#
END

Then rebooted PVE.

Created the VM.

Modified the conf file to be the following:
Code:
root@pve:~# cat /etc/pve/qemu-server/101.conf
bootdisk: scsi0
cores: 1
ide2: none,media=cdrom
memory: 512
name: pci-passthru
numa: 0
ostype: l26
scsi0: local-zfs:vm-101-disk-1,size=2G
scsihw: virtio-scsi-pci
smbios1: uuid=e3edf130-dbbd-4a92-805e-ebad36b457f5
sockets: 1
machine: q35
hostpci0: 26:00
root@pve:~#
END


Boot the VM and observe 1x of the LSI PCI controllers on the VM!


This specific controller doesn't have any disks.
I need to attach the second controller that has the disks.

Stopped the VM.

Modify the conf file to be the following *both* LSI controllers:
Code:
root@pve:~# cat /etc/pve/qemu-server/101.conf
bootdisk: scsi0
cores: 1
ide2: none,media=cdrom
memory: 512
name: pci-passthru
numa: 0
ostype: l26
scsi0: local-zfs:vm-101-disk-1,size=2G
scsihw: virtio-scsi-pci
smbios1: uuid=e3edf130-dbbd-4a92-805e-ebad36b457f5
sockets: 1
machine: q35
hostpci0: 26:00;27:00
root@pve:~#
END


Starting the VM fails with this:
Code:
kvm: -device vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio: Hardware reports invalid configuration, MSIX PBA outside of specified BAR
kvm: -device vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10: Device initialization failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=e3edf130-dbbd-4a92-805e-ebad36b457f5' -name pci-passthru -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:302b94ebab' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-101-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -machine 'type=q35'' failed: exit code 1
END

Revert the conf to only 'hostpci0: 26:00' and it boots.

Add the second card and it won't start.

Set the conf to only the second card with the disks 'hostpci0: 27:00' and get this same error:
Code:
kvm: -device vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio: Hardware reports invalid configuration, MSIX PBA outside of specified BAR
kvm: -device vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10: Device initialization failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=e3edf130-dbbd-4a92-805e-ebad36b457f5' -name pci-passthru -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=27:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:302b94ebab' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-101-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -machine 'type=q35'' failed: exit code 1
END

I've also tried 'hostpci0: 27:00,pcie=1' and it would not boot due to the same error.

Configured it to 'hostpci0: 27:00,pcie=1,driver=vfio' and the VM boots!
Where is the card though? It's not showing during the VM boot process...

Tested the first card with the same config: hostpci0: 27:00,pcie=1,driver=vfio
The VM boots but the card is not showing during VM boot.

This message:
Code:
vfio: Hardware reports invalid configuration, MSIX PBA outside of specified BAR

I can't find anything beyond developer email threads. Nothing concerning what should be done. Hence posting here.

How can I pass-through the LSI controller card with the disks?
How can I pass-through both LSI controller cards?

Please let me know if any additional data is necessary.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!