PCI passthrough LSI 9300-8i to OmniOS

K

kn2mUc

New Member
Jan 30, 2016
23
0
1
42
Im having some issues passing through a LSI 9300-8i HBA to Nappit running under OmniOS.
The cards do not have bootable ROMs installed. I followed the how-to. The 2 connected SSDs are visible in Nappit VM but when trying to create a pool I see the following errors before Napp-it reports an error.

Code: 
Dec 9 19:48:46 nappit scsi; WARNING: /pci@0.0/pci1000,30e0@10 (mph_sas0):
Dec 9 19:48:46 nappit    MPT Firmware Fault, code: 2622

Heres the section from the lspci -vnn
Code: 
01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
    Subsystem: LSI Logic / Symbios Logic SAS9300-8i [1000:30e0]
    Flags: bus master, fast devsel, latency 0, IRQ 36
    I/O ports at e000 [disabled] [size=256]
    Memory at fb840000 (64-bit, non-prefetchable) [size=64K]
    Memory at fb800000 (64-bit, non-prefetchable) [size=256K]
    Expansion ROM at fb700000 [size=1M]
    Capabilities: [50] Power Management version 3
    Capabilities: [68] Express Endpoint, MSI 00
    Capabilities: [a8] MSI: Enable+ Count=1/1 Maskable+ 64bit+
    Capabilities: [c0] MSI-X: Enable- Count=96 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [1e0] #19
    Capabilities: [1c0] Power Budgeting <?>
    Capabilities: [190] #16
    Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
    Kernel driver in use: vfio-pci

02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
    Subsystem: LSI Logic / Symbios Logic SAS9300-8i [1000:30e0]
    Flags: fast devsel, IRQ 36
    I/O ports at d000 [disabled] [size=256]
    Memory at fb640000 (64-bit, non-prefetchable) [disabled] [size=64K]
    Memory at fb600000 (64-bit, non-prefetchable) [disabled] [size=256K]
    Expansion ROM at fb500000 [disabled by cmd] [size=1M]
    Capabilities: [50] Power Management version 3
    Capabilities: [68] Express Endpoint, MSI 00
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [c0] MSI-X: Enable- Count=96 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [1e0] #19
    Capabilities: [1c0] Power Budgeting <?>
    Capabilities: [190] #16
    Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
    Kernel driver in use: vfio-pci

Code: 
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000007E1C6F70 0000CC (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.061930] DMAR: Host address width 46
[    0.061931] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.061935] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.061936] DMAR: RMRR base: 0x0000007f235000 end: 0x0000007f243fff
[    0.061937] DMAR: ATSR flags: 0x0
[    0.061938] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.061939] DMAR-IR: IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.061940] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
[    0.061941] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.061942] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.062126] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.785142] DMAR: dmar0: Using Queued invalidation
[    0.785155] DMAR: Setting RMRR:
[    0.785172] DMAR: Setting identity map for device 0000:00:1a.0 [0x7f235000 - 0x7f243fff]
[    0.785193] DMAR: Setting identity map for device 0000:00:1d.0 [0x7f235000 - 0x7f243fff]
[    0.785205] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.785213] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.785223] DMAR: Intel(R) Virtualization Technology for Directed I/O

tried config with both 'machine: q35' and without.

Code: 
balloon: 0
bootdisk: ide0
cores: 2
cpu: host
ide0: local-zfs:vm-114-disk-1,size=32G
ide2: none,media=cdrom
memory: 16384
name: napp
net0: virtio=DE:DC:DC:5E:6E:B3,bridge=vmbr20
numa: 0
ostype: solaris
scsihw: virtio-scsi-pci
smbios1: uuid=c34bd90a-5c09-4ff5-8629-81ba7b8c3194
sockets: 1
hostpci0: 01:00.0
hostpci1: 02:00.0

I see this illegal PCI request error on booting the VM

proxmox-omnioserror.png

And finally heres the lspci output from the proxmox VM

Code: 
Last login: Sat Dec 10 12:45:16 2016 on console
OmniOS 5.11     omnios-r151020-b5b8c75  November 2016
root@napp:/root# lspci -vnn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Flags: fast devsel

00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Flags: bus master, medium devsel, latency 0

00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010] (prog-if 80 [Master])
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Flags: bus master, medium devsel, latency 0
        I/O ports at e240

00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc QEMU Virtual Machine [1af4:1100]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        I/O ports at e200

00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Flags: medium devsel, IRQ 9

00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8] (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc QEMU Virtual Machine [1af4:1100]
        Flags: bus master, fast devsel, latency 0
        Memory at fa000000 (32-bit, prefetchable)
        Memory at feaf0000 (32-bit, non-prefetchable)
        Expansion ROM at feac0000 [disabled]

00:10.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: LSI Logic / Symbios Logic SAS9300-8i [1000:30e0]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        I/O ports at e000 [disabled]
        Memory at fead0000 (64-bit, non-prefetchable)
        Memory at fea00000 (64-bit, non-prefetchable)
        Expansion ROM at fe800000 [disabled]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [a8] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [c0] MSI-X: Enable- Count=96 Masked-

00:11.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: LSI Logic / Symbios Logic SAS9300-8i [1000:30e0]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        I/O ports at e100 [disabled]
        Memory at feae0000 (64-bit, non-prefetchable)
        Memory at fea40000 (64-bit, non-prefetchable)
        Expansion ROM at fe900000 [disabled]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [a8] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [c0] MSI-X: Enable- Count=96 Masked-

00:12.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
        Subsystem: Red Hat, Inc Device [1af4:0001]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        I/O ports at e220
        Memory at feaf1000 (32-bit, non-prefetchable)
        Memory at fc000000 (64-bit, prefetchable)
        Expansion ROM at fea80000 [disabled]
        Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg

00:1e.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 10
        Memory at feaf2000 (64-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: fe600000-fe7fffff
        Prefetchable memory behind bridge: 00000000fca00000-00000000fcbfffff
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 01
        Capabilities: [40] Hot-plug capable

00:1f.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11
        Memory at feaf3000 (64-bit, non-prefetchable)
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: fe400000-fe5fffff
        Prefetchable memory behind bridge: 00000000fc800000-00000000fc9fffff
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 02
        Capabilities: [40] Hot-plug capable
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back