Hi all,
I'm trying to get PCIe passthrough running with proxmox 4.0.57 and an IBM M1115 controller crossflashed to the LSI IT firmware. The Mainboard is a Supermicro X10SLL-F with a Xeon E3-1226 v3.
I used rockstor (centos) at first and tried setting up a btrfs pool, but installed debian 8 to see if I get the same problems in another "clean" OS.
I get the controller passed through to the VM and I can see the HDDs (4x 6TB WD red), but I can't really use the HDDs:
fdisk output, disks loose their partitioning at random (they come "back" after a reboot, but mounting still not possible)
fdisk output some 20 minutes earlier:
but lsblk still shows the partitions:
Creating a filesystem:
qm monitor 100
Proxmox log output
vm 100.conf
The controller is added via the quemu/vm conf file: hostpci0: 01:00,pcie=1,driver=vfio and all the necessary options are set according to https://pve.proxmox.com/wiki/Pci_passthrough1 and https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
I'm at a total loss here, everything seems fine. Even no errors in any logfiles (host and vm). Hopefully someone has an idea how to tackle this.
Cheers and thanks for reading this wall of text
I'm trying to get PCIe passthrough running with proxmox 4.0.57 and an IBM M1115 controller crossflashed to the LSI IT firmware. The Mainboard is a Supermicro X10SLL-F with a Xeon E3-1226 v3.
I used rockstor (centos) at first and tried setting up a btrfs pool, but installed debian 8 to see if I get the same problems in another "clean" OS.
I get the controller passed through to the VM and I can see the HDDs (4x 6TB WD red), but I can't really use the HDDs:
- Accessing the smart values is very spotty at best
- "device is read only" errors from rockstor when adding a btrfs pool
- mount errors (wrong fs type, bad option, bad superblock on /dev/sda1, missing codepage or helper program, or other error) for either xfs and ext3 filesystems
- disks loose their partitioning seemingly at random/after some time. Could not reproduce this
- smartctl not working as expected (tried different options -d sat etc, no change), inside rockstor smart values would show in 1 of 10 tries
Code:
root@rockstor:~# smartctl -i /dev/sdasmartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFRX-68MYMN1
Serial Number: WD-WX21D15A74Y0
LU WWN Device Id: 5 0014ee 2b6886b68
Firmware Version: 82.00A82
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5700 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Dec 3 10:21:21 2015 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
root@rockstor:~# smartctl -i /dev/sda
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
Short INQUIRY response, skip product id
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.fdisk output, disks loose their partitioning at random (they come "back" after a reboot, but mounting still not possible)
Code:
root@rockstor:~# fdisk -l
Disk /dev/vda: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb850e335
Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 64286719 64284672 30.7G 83 Linux
/dev/vda2 64288766 67106815 2818050 1.4G 5 Extended
/dev/vda5 64288768 67106815 2818048 1.4G 82 Linux swap / Solaris
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdb: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sda: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
The backup GPT table is corrupt, but the primary appears OK, so that will be used.
Disk /dev/sdd: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0EDA213C-9ACE-4751-ABFC-5DF85E4F918D
Device Start End Sectors Size Type
/dev/sdd1 2048 11721045134 11721043087 5.5T Linux filesystem
Code:
root@rockstor:~# fdisk -l
Disk /dev/vda: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb850e335
Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 64286719 64284672 30.7G 83 Linux
/dev/vda2 64288766 67106815 2818050 1.4G 5 Extended
/dev/vda5 64288768 67106815 2818048 1.4G 82 Linux swap / Solaris
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E1F3C335-1D64-4F29-AEE2-EA14F2149B63
Device Start End Sectors Size Type
/dev/sdc1 2048 11721045134 11721043087 5.5T Linux filesystem
Disk /dev/sdb: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 70E83063-BA1F-446B-B61B-BA14D5D512CE
Device Start End Sectors Size Type
/dev/sdb1 2048 11721045134 11721043087 5.5T Linux filesystem
Disk /dev/sda: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D0DDBDFB-6DC9-4990-BF21-46838C3F8628
Device Start End Sectors Size Type
/dev/sda1 2048 11721045134 11721043087 5.5T Linux filesystem
Disk /dev/sdd: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0EDA213C-9ACE-4751-ABFC-5DF85E4F918D
Device Start End Sectors Size Type
/dev/sdd1 2048 11721045134 11721043087 5.5T Linux filesystem
Code:
root@rockstor:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 5.5T 0 disk
└─sda1 8:1 0 5.5T 0 part
sdb 8:16 0 5.5T 0 disk
└─sdb1 8:17 0 5.5T 0 part
sdc 8:32 0 5.5T 0 disk
└─sdc1 8:33 0 5.5T 0 part
sdd 8:48 0 5.5T 0 disk
└─sdd1 8:49 0 5.5T 0 part
sr0 11:0 1 247M 0 rom
vda 254:0 0 32G 0 disk
├─vda1 254:1 0 30.7G 0 part /
├─vda2 254:2 0 1K 0 part
└─vda5 254:5 0 1.4G 0 part [SWAP]Creating a filesystem:
Code:
root@rockstor:~# mkfs.ext3 /dev/sdb1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 1465130385 4k blocks and 183144448 inodes
Filesystem UUID: 18f2512e-f6bc-47f1-850b-0cec02534612
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
root@rockstor:~# mount /dev/sda1 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
root@rockstor:~# fsck /dev/sda1
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda1
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
root@rockstor:~# e2fsck -b 8193 /dev/sda
e2fsck 1.42.12 (29-Aug-2014)
e2fsck: Bad magic number in super-block while trying to open /dev/sda
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
Code:
root@rockstor:~# lspci -nnk
01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: LSI Logic / Symbios Logic Device [1000:3020]
Kernel driver in use: mpt2sasqm monitor 100
Code:
info pciBus 1, device 0, function 0:
SAS controller: PCI device 1000:0072
IRQ 10.
BAR0: I/O at 0x7000 [0x70ff].
BAR1: 64 bit memory at 0xfe8c0000 [0xfe8c3fff].
BAR3: 64 bit memory at 0xfe880000 [0xfe8bffff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0007fffe].id "hostpci0"Proxmox log output
Code:
Dec 2 22:55:28 intrepid kernel: [ 680.200164] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)vm 100.conf
Code:
balloon: 2048bootdisk: virtio0
cores: 4
cpu: host
ide2: local:iso/debian-8.1.0-amd64-netinst.iso,media=cdrom,size=247M
machine: q35
hostpci0: 01:00.0,pcie=1
memory: 8192
name: rockstor
net0: e1000=56:51:10:FE:F0:A5,bridge=vmbr0
numa: 0
ostype: l26
smbios1: uuid=4fc0047a-ac89-45f3-9d02-2c1c251364ab
sockets: 1
unused0: local:100/vm-100-disk-1.qcow2
virtio0: local:100/vm-100-disk-2.qcow2,cache=writeback,size=32G
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 rootdelay=10 scsi_mod.scan=sync"
Code:
# /etc/modules: kernel modules to load at boot time.#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
#pci_stub
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
kvm
kvm_intel
Code:
/etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
#pci_stub ids=1000:0072
vfio-pci ids=1000:0072
Code:
dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x00000000DDA63A50 000080 (v01 INTEL BDW 00000001 INTL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.028070] DMAR: Host address width 39
[ 0.028071] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.028077] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.028078] DMAR: RMRR base: 0x000000df697000 end: 0x000000df6a5fff
[ 0.028080] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0
[ 0.028080] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[ 0.028081] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.028082] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.028226] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.557899] DMAR: No ATSR found
[ 0.557961] DMAR: dmar0: Using Queued invalidation
[ 0.557968] DMAR: Setting RMRR:
[ 0.557977] DMAR: Setting identity map for device 0000:00:14.0 [0xdf697000 - 0xdf6a5fff]
[ 0.557998] DMAR: Setting identity map for device 0000:00:1a.0 [0xdf697000 - 0xdf6a5fff]
[ 0.558015] DMAR: Setting identity map for device 0000:00:1d.0 [0xdf697000 - 0xdf6a5fff]
[ 0.558027] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.558032] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.558041] DMAR: Intel(R) Virtualization Technology for Directed I/OI'm at a total loss here, everything seems fine. Even no errors in any logfiles (host and vm). Hopefully someone has an idea how to tackle this.
Cheers and thanks for reading this wall of text
