PCI Passthrough with 1068e onboard SAS controller

[jussuf]

Just fixed my upload scripts - please can you test again?

That's better now, thanks.

 

[jussuf]

Short howto for building custom kernel.

1. Download latest kernel source package (previous one with the same name had problems).
2. tar xzf pve-kernel-2.6.32_2010-05-25.tar.gz
3. cd pve-kernel-2.6.32_2010-05-25
4. make config-2.6.32
5. Make changes to config-2.6.32 (eg. replace '# CONFIG_DMAR is not set' with 'CONFIG_DMAR=Y')
6. make

 

[jussuf]

If you really want to waste our time, I can burn and send you a CD with all sources included.

No need to do that, the older kernel is not really useful anymore. But as I stated before, the previously available pve-kernel-2.6.32_2010-02-04.tar.gz did not build into a correct 2.6.32-1-pve binary package. If you pinpoint git checkout to a specific branch, it should be quite easy to make simple and small source distribution, that compiles and builds correct binary. Seems that source hasn't really interested anyone before, people are happy using the binary distribution and you shouldn't worry much about the bandwidth

 

[dietmar]

If you pinpoint git checkout to a specific branch, it should be quite easy to make simple and small source distribution, that compiles and builds correct binary.

I already do that - it is a bug if it does not work (feel free to send a patch).

 

[dietmar]

CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y

Do we need CONFIG_INTR_REMAP?

 

[jussuf]

Do we need CONFIG_INTR_REMAP?

My initial test ran without it, but KVM howto tells that it's required. Maybe some devices don't need it and I was just lucky.
Edit: Sorry, it's marked optional in the howto, missed the 'optional' part at first.

 

[VGusev2007]

Thank a lot. HOW-TO is GOOD!

So, I have made kernel, it's work so-so. I'm boot into that.

I make it without CONFIG_INTR_REMAP.

Please hint me about PCI:

My USB controller is:

lspci
01:04.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
01:04.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
01:04.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 63)

lspci -n
01:04.0 0c03: 1106:3038 (rev 61)
01:04.1 0c03: 1106:3038 (rev 61)
01:04.2 0c03: 1106:3104 (rev 63)

How may I attach it to my VM?

I unbound it from my real PC, the next I add the following into VM config file:

args: -pcidevice host=01:04.0(,host=01:04.1)

Next I do:

proxmox-52:~# qm start 101
device: 01:04.0: driver="pci-assign" host="01:04.0"

But it's not work. Start VM is fine! VM not detect a new hardware.

IOMMU output is more now!

proxmox-52:~# dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 000000007f79e0c0 00140 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
DMAR: Host address width 40
DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020fe
DMAR: RMRR base: 0x000000000e5000 end: 0x000000000e7fff
DMAR: RMRR base: 0x0000007f7ed000 end: 0x0000007f7fffff
DMAR: ATSR flags: 0x0
IOMMU 0xfbffe000: using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.0 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.2 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.3 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.2 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.3 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xe5000 - 0xe8000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]

I'll recompile kernel with new option, I hope it will works.

Please help me, how to work with my real USB controller into VM.

 

[VGusev2007]

I have recompile my kernel with:

CONFIG_INTR_REMAP=y
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR=y

It's ok.

I run these command:

proxmox-52:~# echo "1106 3038" > /sys/bus/pci/drivers/pci-stub/new_id
proxmox-52:~# echo 0000:01:04.0 > /sys/bus/pci/devices/0000\:01\:04.0/driver/unbind
proxmox-52:~# echo 0000:01:04.1 > /sys/bus/pci/devices/0000\:01\:04.1/driver/unbind
proxmox-52:~# echo 0000:01:04.2 > /sys/bus/pci/devices/0000\:01\:04.2/driver/unbind

#qm monitor 101
qm> pci_add pci_addr=auto host host=01:04.0
Error initializing device pci-assign
failed to add host=01:04.0

Please help me!

# dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 000000007f79e0c0 00140 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
DMAR: Host address width 40
DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020fe
DMAR: RMRR base: 0x000000000e5000 end: 0x000000000e7fff
DMAR: RMRR base: 0x0000007f7ed000 end: 0x0000007f7fffff
DMAR: ATSR flags: 0x0
IOMMU 0xfbffe000: using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.0 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.2 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.3 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0x7f7ed000 - 0x7f800000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.2 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.3 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xe5000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xe5000 - 0xe8000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]

VT-d and other options is enabled in the BIOS.

 

[jussuf]

Please help me, how to work with my real USB controller into VM.

Well, you just have to try with different devices. I suggest to first try with a network card, this usually works best with PCI passthrough. Some devices may not work at all in passthrough mode, this is highly dependant on hardware and may involve some magic

If you want to pass some USB devices to the VM, maybe hostusb option works better.

 

[VGusev2007]

Well, you just have to try with different devices. I suggest to first try with a network card, this usually works best with PCI passthrough. Some devices may not work at all in passthrough mode, this is highly dependant on hardware and may involve some magic

If you want to pass some USB devices to the VM, maybe hostusb option works better.

I'm upset. You are right! I try a sound card and USB controller - I have installed it into a PCI slot (Not a PCI-E!). - This is not working. I have some error message.

I have mother board INTEL 5xxx.

Just I setup a network card into a PCI-E slot, it is working fine! No errors message. VM is detect a new hardware success.

If you want to pass some USB devices to the VM, maybe hostusb option works better.

It's no working with hostusb option. I have a USB->to->COM port controller device. When I connect it, VM is detect a new hardware. I setup a driver for this one. It is correct, but I can't read data from it. It not working.

A HASP works fine.

Any ideas? I'll bue a USB Pci-e controller maybe, but i don't have a free PCI-e slot, and it's cheap.

I'll setup a network card into a PCI slot (not a PCI-e) tomorow. I'll write to here about how it works.

I have read about:

VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

What means that? The BIOS of my mother board say me about PCI bridge, but I'm not clean about it.

I cry. If you have any ideas please tell me about this.

 

[VGusev2007]

and it's cheap.

Fix: it is expensive

 

[VGusev2007]

I have installed a NIC into a PCI slot (NOT a PCI-e), it not working.

I think PCI Passthrough is not passible. It will work with PCI-e ONLY.

It is shit...

I cry.

 

[dietmar]

It is shit...

I cry.

Well, most people use virtualization just to avoid problems with hardware. Using pci pass-through brings those problems back to you. So think twice - do you really need it.

 

[VGusev2007]

Well, most people use virtualization just to avoid problems with hardware. Using pci pass-through brings those problems back to you. So think twice - do you really need it.

Yes. I understand about it. I want to fuse several dedicated server into VM. Oh... Intel-VD is not a solution for all problem with hardware.

Thank a lot for You and jussuf. I'll use it without USB real port. I'll fuse other dedicated servers. I'll reconfigure kernel because I know how to do it. THANK!