PCI Passthrough with 1068e onboard SAS controller

  • Thread starter Thread starter ben001
  • Start date Start date
B

ben001

Guest
I am trying to pass a sas controller to a osol vm, has anyone got this working?

I followed the instructions here http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM to build the kernel with the correct options. I don't see anything in dmesg about iommu but i do see the following for DMAR. This is a supermicro X8ST3-F motherboard with the latest bios and vt-d enabled.

Code: 
dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 00000000df79e0d0 00120 (v01    AMI  OEMDMAR 00000001 MSFT 00000097)
I get the following errors when i boot osol build 133

Code: 
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
unknown ioc_status = 0
scsi_state = 0, transfer count = 0, scsi_status = 0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
NULL command returnedas address reply int slot 0, function:0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
Disconnected command timeout for Target 0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
NULL command returnedas address reply int slot 0, function:0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
NULL command returnedas address reply int slot 0, function:0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
NULL command returnedas address reply int slot 0, function:0
WARNING: /pci@0,0/pci15d9,5@5 (mpt0):
NULL command returnedas address reply int slot 0, function:0
etc
and this error if i try to pass the controller into a windows 2008R2 vm (with osol shutdown), there are no devices on port 0 of the controller.
Code: 
The driver detected a controller error on \Device\RaidPort0.
and finally this error and stalls when trying to boot up a gentoo vm
Code: 
scanning for mptsas...scsi_transport_sas,
dmesg shows shows that i have successfully unbound the device and bound it to the stub driver with this code
Code: 
echo "[FONT=Courier New]1000[/FONT] [FONT=Courier New]0058[/FONT]" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:05:00.0 > /sys/bus/pci/devices/0000:05:00.0/driver/unbind
echo 0000:05:00.0 > /sys/bus/pci/drivers/pci-stub/bind
Code: 
dmesg
mptsas 0000:05:00.0: PCI INT A disabled
pci-stub 0000:05:00.0: claimed by stub
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
device vmtab101i0 entered promiscuous mode
vmbr0: port 2(vmtab101i0) entering forwarding state
pci-stub 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci-stub 0000:05:00.0: restoring config space at offset 0x1 (was 0x40100000, writing 0x40100143)
  alloc irq_desc for 45 on node -1
  alloc kstat_irqs on node -1
pci-stub 0000:05:00.0: irq 45 for MSI/MSI-X
pci-stub 0000:05:00.0: irq 45 for MSI/MSI-X
kvm:2872 freeing invalid memtype fadfe000-fadff000
vmtab101i0: no IPv6 routers present
pveversion
Code: 
pve-manager: 1.5-8 (pve-manager/1.5/4674)
running kernel: 2.6.32-1-pve
proxmox-ve-2.6.32: 1.5-4
pve-kernel-2.6.32-1-pve: 2.6.32-4
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.11.1-2
ksm-control-daemon: 1.0-3
lspci

Code: 
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
08:04.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)
vm configuration
Code: 
name: Solaris
ide2: local:iso/osol-dev-133-x86.iso,media=cdrom
bootdisk: ide0
ostype: other
memory: 2048
sockets: 1
boot: dc
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan0: e1000=5A:FE:B2:5A:24:9F
args: -pcidevice host=05:00.0
onboot: 0
cores: 4
Am i missing something?
 
I suggest you test again as soon as we released pve-qemu-kvm 0.12.3
 
is pve-qemu-kvm 0.12.3 something that has already been released and i just cant find it?
 
not released, currently testing in our test lab.
 
do you happen to know when this might be released to the masses? few days or few weeks?
 
Current proxmox kernel 2.6.32-2-pve lacks DMAR/IOMMU support, at least for Intel platform.
To use PCI passthrough, please add following parameters to the kernel configuration:
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y

I have tested this successfully with L5520 CPU and 5520 chipset. Hopefully it will be included in next Proxmox kernel release.
 
jussuf said:
Current proxmox kernel 2.6.32-2-pve lacks DMAR/IOMMU support, at least for Intel platform.
To use PCI passthrough, please add following parameters to the kernel configuration:
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y

I have tested this successfully with L5520 CPU and 5520 chipset. Hopefully it will be included in next Proxmox kernel release.
Click to expand...

Hi, I need your help. Pleas tell me How to do it. Please post short how-to with it.

Where can I take src for 2.6.32-2-pve kernel?
 
VGusev2007 said:
Where can I take src for 2.6.32-2-pve kernel?
Click to expand...

That's a question I asked in another thread also, and Proxmox staff haven't answered it yet. Source should be at ftp://download.proxmox.com/sources/ but pve-kernel-2.6.32_2010-02-04.tar.gz builds into 2.6.32-1-pve kernel and 2.6.32-2-pve source is missing. For now, I used what's available and just changed the configuration. It's supposed to be quite straightforward -- download, unpack and run make.

Unfortunately the source package, when unmodified, does currently not build an identical kernel to the distributed binary. The catch is, that this 'source' does not contain actual kernel source, but depends on Ubuntu kernel repository to which it adds only proxmox-specific changes. Ubuntu repository contents have evolved in time and some configuration patches even don't apply cleanly, so you either have to hack it together on current sources OR do a checkout from the exact date when actual proxmox binary packages where built.
While otherwise Proxmox VE is a great product, I am somewhat dissapointed for the source issue. When I started to look for sources, I was hoping for debian source packages and/or a public git tree, but found only some outdated code drops.
 
jussuf said:
I am somewhat dissapointed for the source issue. When I started to look for sources, I was hoping for debian source packages and/or a public git tree, but found only some outdated code drops.
Click to expand...

Just uploaded the latest sources.
 
cd linux-2.6.32; patch -p1 <../disable-fsam7400.diff
patching file ubuntu/Makefile
Hunk #1 FAILED at 5.
1 out of 1 hunk FAILED -- saving rejects to file ubuntu/Makefile.rej
make: *** [linux-2.6.32/README] Error 1
Click to expand...


install: cannot create regular file `tmp/lib/modules/2.6.32-1-pve/kernel/drivers/block/aoe/aoe.ko': No such file or directory
make: *** [data] Error 1
Click to expand...
I have these error when I make kernel.
 
Last edited:
dietmar said:
Please run 2.6.32-2-pve - build should work then.
Click to expand...
I need help.

I make it by:
cd pve-kernel-2.6.32_2010-05-25
edit file: config-2.6.32.diff (add lines: CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y)
Like this:
--- config-2.6.32.org 2010-01-15 09:46:50.000000000 +0100
+++ config-2.6.32 2010-01-15 11:21:17.000000000 +0100
@@ -820,21 +820,9 @@
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y
CONFIG_DONGLE=y
CONFIG_DOUBLEFAULT=y
CONFIG_DRAGONRISE_FF=y
-CONFIG_DRM=m
Click to expand...

run:
make clean
make

Please tell how to do it correctly.

I need help. I'm not clean about make kernel.
 
dietmar said:
Just uploaded the latest sources.
Click to expand...

Are you kidding me?
This pve-kernel-2.6.32_2010-05-25.tar.gz package contains exactly the same source as previous pve-kernel-2.6.32_2010-02-04.tar.gz (which, strangely, is not available any more).
Code: 
diff -urN pve-kernel-2.6.32_2010-02-04/ pve-kernel-2.6.32_2010-05-25/
gives zero differences.
Also, there is KREL=1 in Makefile, so it certainly doesn't compile into pve-kernel-2.6.32-2-pve. Neither does it correctly reproduce a pve-kernel-2.6.32-1-pve package, as make fails with
Code: 
cd linux-2.6.32; patch -p1 <../disable-fsam7400.diff
patching file ubuntu/Makefile
Hunk #1 FAILED at 5.
1 out of 1 hunk FAILED -- saving rejects to file ubuntu/Makefile.rej
make: *** [linux-2.6.32/README] Error 1
Even after patching it up into a compilable state, it doesn't compile into same kernel that is distributed in binary package. Distributed binary package is pve-kernel-2.6.32-1-pve_2.6.32-4_amd64.deb but Makefile contains PKGREL=5 and as all patches don't apply, I'm pretty sure that the source has changed.

As Proxmox VE is licenced under GPLv2, you have to provide source for all distributed binaries. And not just any random source, but exactly the source that builds into distributed binaries. I haven't looked into other packages, but kernel source distribution certainly has problems.
 
With the current unmodify kernel I have this error:

My

lspci -n
01:04.0 0c03: 1106:3038 (rev 61)
01:04.1 0c03: 1106:3038 (rev 61)
01:04.2 0c03: 1106:3104 (rev 63)
Click to expand...

It is my usb controller:

lspci
01:04.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
01:04.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
01:04.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 63)
Click to expand...

I attempt to unbind it by:
echo "1106 3038" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:04.0 > /sys/bus/pci/devices/0000\:01\:04.0/driver/unbind and some with 01:04.1 and 01:04.2
Click to expand...

My dmesg is:
uhci_hcd 0000:01:04.0: remove, state 4
usb usb10: USB disconnect, address 1
uhci_hcd 0000:01:04.0: USB bus 10 deregistered
uhci_hcd 0000:01:04.0: PCI INT A disabled
device vmtab101i0 entered promiscuous mode
uhci_hcd 0000:01:04.1: USB bus 11 deregistered
uhci_hcd 0000:01:04.1: PCI INT B disabled
ehci_hcd 0000:01:04.2: remove, state 4
usb usb3: USB disconnect, address 1
ehci_hcd 0000:01:04.2: USB bus 3 deregistered
ehci_hcd 0000:01:04.2: PCI INT C disabled
Click to expand...

IOMMU:
dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 000000007f79e0c0 00140 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
Click to expand...

pveversion --verbose
pve-manager: 1.5-9 (pve-manager/1.5/4728)
running kernel: 2.6.32-2-pve
proxmox-ve-2.6.32: 1.5-7
pve-kernel-2.6.32-2-pve: 2.6.32-7
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-14
pve-firmware: 1.0-5
libpve-storage-perl: 1.0-13
vncterm: 0.9-2
vzctl: 3.0.23-1pve11
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.4-1
ksm-control-daemon: 1.0-3
Click to expand...

My VM config:
name: xp
ide2: local:iso/Windows_XP_Pro_SP3_Russian_-_Updates_MAY-2010_+_SATA-RAID.iso,media=cdrom
vlan0: virtio=B2:47:64:9E:2E:F3
bootdisk: virtio0
ostype: wxp
virtio0: local:101/vm-101-disk-1.raw
memory: 512
sockets: 1
boot: cd
freeze: 0
cpuunits: 8000
acpi: 1
kvm: 1
args: -pcidevice host=01:04.0
Click to expand...

When I try for start my VM from command line it fail:
qm start 101
device: 01:04.0: driver="pci-assign" host="01:04.0"
No IOMMU found. Unable to assign device "01:04.0"
Failed to deassign device "01:04.0" : Invalid argument
Error initializing device pci-assign
start failed: .....
Click to expand...

I'm sorry English is not my native language.
 
jussuf said:
As Proxmox VE is licenced under GPLv2, you have to provide source for all distributed binaries. And not just any random source, but exactly the source that builds into distributed binaries.
Click to expand...

We always release the sources a few days after we update the repository. To save bandwidth, we try to make the packages as short as possible. If you really want to waste our time, I can burn and send you a CD with all sources included.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom