Problems with Illumos based VM & passed through LSI Controller

[mauorrizze]

Trying to install a ZFS capable Storage VM, I'm running into problems with this combination:
PVE (2.2 pve & pvetest) + Illumos based VM (OpenIndiana, OmniOS) + pass through LSI Controller (IBM 1015 or BR10i cross flashed with LSI-IT firmware)

If I change one of these things, it works, be it
PVE + FreeBSD + LSI
ESXi + Illumos + LSI
or PVE + Illumos without LSI, e.g. onboard

The error I get is that soon after two Warnings appear, the Guest automatically reboots.

WARNING: devi_bus information is not set for isa0.
WARNING: devi_bus information is not set for vgatext0.

Google takes me only to the responsible source code, which has something to do with PCIe error handling and FMA capabilities.

My hardware: Intel i5-2400 @ Asus P8B WS (C206), may be its BIOS has a faulty vt-d configuration?
My extra line in the config:

hostpci0: 02:00.0
#or
args: -device pci-assign,romfile=,host=02:00.0 #to prevent loading the option Rom, which does not at all or partly work depending on the firmware version

The mtp2sas or mtpsas modules are blacklisted, so passing the device through works in the beginning.

Currently I'm using ESXi again but it would be great if anyone has ideas that bring me nearer to a working PVE solution.

 

[churnd]

Having the same issue myself currently. Any solutions found to this? I think it is an OmniOS specific issue.

 

[mauorrizze]

Unfortunately, no. All I can confirm is that this problems occurs with the IBM/LSI Controller when combining a Linux/KVM host with a Illumnos-based guest OS. Haven't found such combination that works (Gentoo with different kernels, several PVE versions, older and newer OmniOS, OI).
Which controller do you use?

I believe I can rule out the faulty BIOS/UEFI: other Boards (ASRock B75, MSI Z77?-not sure on that) are also affected.

As a working alternative I can recommend PVE with a linux guest and ZFS on Linux. After a board change and resulting vt-d-passthrough problems I even returned to use ZFS on the host (installed PVE ontop of Debian).
Maybe next year I can test a AMD/IOMMU based desktop-board.

 

[churnd]

Unfortunately, no. All I can confirm is that this problems occurs with the IBM/LSI Controller when combining a Linux/KVM host with a Illumnos-based guest OS. Haven't found such combination that works (Gentoo with different kernels, several PVE versions, older and newer OmniOS, OI).
Which controller do you use?

I believe I can rule out the faulty BIOS/UEFI: other Boards (ASRock B75, MSI Z77?-not sure on that) are also affected.

As a working alternative I can recommend PVE with a linux guest and ZFS on Linux. After a board change and resulting vt-d-passthrough problems I even returned to use ZFS on the host (installed PVE ontop of Debian).
Maybe next year I can test a AMD/IOMMU based desktop-board.

That solution is in the back of my mind, but I would rather keep at trying to fix this for the time being. I would use it for a lot more than just storage for the host... AFP/SMB shares to other computers in the house for example. Not sure if I want to extend Proxmox's role beyond a hypervisor.

 

[gutleib]

I can only confirm that the problem exists.

AMD Opteron 6300, IOMMU, LSI 9207+LSI9211, kernel 3.12

reboot cycle

 

[gutleib]

update: any passthrough, tried intel nic, - the same reboot cycle

 

[churnd]

update: any passthrough, tried intel nic, - the same reboot cycle

I think any type of kvm passthrough is going to break Illumos guests. I've tried multiple types of solaris guests going back to opensolaris, including NexentaStor, & all had the same result.