Proxmox VE 4.2 pci passthrough LSI 3008 (mpt3sas)

[hoppel118]

Hello guys,

at the moment I am virtualizing my openmediavault to a kvm. My hardware specs:

Mainboard: Supermicro X11SSH-CTF
CPU: Intel Xeon E3-1240Lv5 4x 2.10GHz So.1151 TRAY
RAM: 64GB DDR4 ECC UDIMM
Onboard-SAS-Controller: LSI SAS3008 PCI-Express (with 8x4TB WD Red in a Eaid-Z2)

On the host I am Running Proxmox 4.2 with kernel 4.4.19-1-pve.

So I want to passthrough my onboard "LSI SAS-Controller" and did it the way it is written in the following article:

https://pve.proxmox.com/wiki/Pci_passthrough

In detail I did the following:

1. Changed to: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on" in "/etc/default/grub", executed "update-grub" and rebooted
2. added the vfio-modules (vfio, vfio_iommu_type1, vfio_pci and vfio_virqfd) to "etc/modules"
3. built the kvm with openmediavault3 (debian jessie 8.6) and installed the pve kernel in the kvm, so both systems (proxmox-host and omv-guest) use the same kernel
4. "dmesg | grep ecap" shows "ecap f050da", the last character is an a, interrupt remapping is supported
5. checked the iommu isolation with "find /sys/kernel/iommu_groups/ -type l", iommu_groups are there
6. located my sas-controller with lspci, the address is: "01:00.0"
7. blacklisted the sas-module on the proxmox-host with "echo 'blacklist mpt3sas' >> /etc/modprobe.d/mpt3sas.conf"
8. depmodded "depmod -aeF /boot/System.map-4.4.19-1-pve"
9. updated initramfs with "update-initramfs -k all -u"
10. rebooted, checked the "mpt3sas" module with lsmod, it's not loaded anymore
    
11. configured my "/etc/pve/qemu-server/vmid.conf" in different ways because I had problems to import my zfs pool to openmediavault3:

At first I added the following lines like you wrote in your article:

machine: q35
hostpci0: 01:00.0,pcie=1

While booting the kvm, I saw the following. lspci showed the sas-controller, but in openmediavault3 I was not able to import the pool.

My syslog was full of scsi and mpt3sas messages: http://pastebin.com/TnTcpEVc

After that I googled around and found the info, that I can use another parameter. So I changed the configuration to the following:

machine: q35
hostpci0: 01:00.0,pcie=1,driver=virtio

But with this parameter I didn't see the sas-controller with lspci anymore.

So I decided to try the following configuration:

machine: q35
hostpci0: 01:00.0

With this configuration it was possible to see the sas-controller with lspci and to import the zfs-pool to openmediavault.

root@omv:~# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Cirrus Logic GD 5446
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.1 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.2 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
06:05.0 SCSI storage controller: Red Hat, Inc Virtio SCSI
06:0c.0 SCSI storage controller: Red Hat, Inc Virtio block device
06:10.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
06:12.0 Ethernet controller: Red Hat, Inc Virtio network device

This is great!

Why do I have to configure the controller without the parameter "pcie=1"?

If I understand your wiki-article right, this parameter should be used for pci-express devices. The onboard-sas-controller uses pci-express.

What limitations can be expected with my configuration?

Sometimes I see the follwoing message in /var/log/messages. http://pastebin.com/QyEPTnG8

Oct  1 03:00:11 omv kernel: [    2.408262] virtio-pci 0000:06:03.0: virtio_pci: leaving for legacy driver

What do you think about this?


When I boot the kvm I see the following fault-state:

You also see this message in the actual syslog: http://pastebin.com/TW93MwS8

Now syslog looks ok all in all in my opinion, except this one fault_state. All other messages in the syslog kvm relating to mpt3sas look like on the bare metal. Great!


By the way, my steps 7,8, 9and 10 are not documented in the wiki-article. Are these steps relevant to do or not?

Thanks in advance for your help!

Greetings Hoppel

 

[MaddinB]

I have similar problems with my X11SSH-CTF board and PVE 4.4 :/

with vm config like this:

machine: q35
hostpci0: 01:00.0,pcie=1

The LSI 3008 SAS Controller constantly resets and system hangs in iowait ...

With no "pcie=1" all seems to be ok, except a great performance drop:
I get only ~50 MB/s write speed to my HDDs on the pass throughed LSI 3008 SAS Controller.

Anyone using a similar setup with pcie passthrough?

 

[adamb]

We are currently passing 4x SAS 3008 cards to a CentOS7 guest.

We also don't use the "pcie=1" option. We do use ovmf bios and machine type q35.

bios: ovmf
boot: cd
bootdisk: scsi0
cores: 24
cpu: host
hostpci0: 08:00.0
hostpci1: 0b:00.0
hostpci2: 81:00.0
hostpci3: 88:00.0
machine: q35
memory: 98304
name: monthvault
net0: virtio=36:63:36:33:35:38,bridge=vmbr0
net1: virtio=62:38:65:30:34:39,bridge=vmbr1
numa: 1
ostype: l26
scsi0: local:100/vm-100-disk-1.raw,size=150G
scsihw: virtio-scsi-pci
smbios1: uuid=5eceeeef-abaa-469c-813a-95eeab1638fe
sockets: 2

Performance has been solid, our only issue in this setup is we can't go above 100G of ram on the VM. Here is a snipit of zpool iostat to give you an idea of our current performance.

[root@monthvault ~]# zpool iostat 1 100 B/s 0.00 % 0.00 % [rcuob/11]
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
M2 108T 37.1T 914 1.28K 84.6M 88.6M
M2 108T 37.1T 1.76K 3.75K 224M 376M
M2 108T 37.1T 1.76K 3.08K 224M 295M
M2 108T 37.1T 1.53K 3.25K 193M 295M
M2 108T 37.1T 1.63K 3.06K 207M 292M
M2 108T 37.1T 1.65K 3.09K 210M 276M
M2 108T 37.1T 1.89K 2.96K 239M 280M
M2 108T 37.1T 1.83K 2.61K 232M 257M
M2 108T 37.1T 1.52K 3.58K 192M 353M
M2 108T 37.1T 1.64K 3.11K 209M 217M

We have been using this same setup since Proxmox 3.

 

[MaddinB]

thx adamb.

I had setup a new test VM with ovmf UEFI.
Until now I have no controller resets anymore, but I have strange errors every 30 seconds:

Message from syslogd@nas at Apr  2 18:21:26 ...
 kernel:[  774.108331] Uhhuh. NMI received for unknown reason 20 on CPU 0.
Message from syslogd@nas at Apr  2 18:21:26 ...
 kernel:[  774.108357] Do you have a strange power saving mode enabled?
Message from syslogd@nas at Apr  2 18:21:26 ...
 kernel:[  774.108375] Dazed and confused, but trying to continue
Message from syslogd@nas at Apr  2 18:21:56 ...
 kernel:[  804.108597] Uhhuh. NMI received for unknown reason 10 on CPU 0.
Message from syslogd@nas at Apr  2 18:21:56 ...
 kernel:[  804.108623] Do you have a strange power saving mode enabled?
Message from syslogd@nas at Apr  2 18:21:56 ...
 kernel:[  804.108641] Dazed and confused, but trying to continue

my actual vm config:
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: local:1000/vm-1000-disk-2.raw,size=128K
hostpci0: 01:00.0,pcie=1
ide2: local:iso/debian-8.7.1-amd64-netinst.iso,media=cdrom
keyboard: de
machine: q35
memory: 32768
name: OMV-UEFU
net0: virtio=1A:92:12:C1:8EF,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:1000/vm-1000-disk-1.raw,size=12G
scsihw: virtio-scsi-pci
smbios1: uuid=a3d2b7d9-b596-4fe2-baee-8aa05c3ff38c
sockets: 1
hostpci0: 01:00.0

Have tested with pcie=1 and without, makes no difference :/

//edit:
finally I have a working setup with proxmox 5.0 beta
(since 24h no NMI error, no controller resets and no performance issues)