Issues with iommu with 3.10 kernel in pvetest repo

mcflym

Active Member
Jul 10, 2013
194
8
38
root@proxmox:/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-4-pve root=/dev/mapper/pve-root ro quiet pcie_acs_override=downstream intel_iommu=on

and still in the same group????
 

XueSheng

Active Member
Feb 18, 2014
45
0
26
Did you apply the pcie_acs_override patch to the kernel? The commandline itself is useless if the kernel doesn't provide this function. If you don't want to build the kernel yourself, I can provide you the latest git kernel with patch applied.
 

mcflym

Active Member
Jul 10, 2013
194
8
38
argh... of course i did NOT :cool:

if you can provide the latest kernel with the patch please feel free to share it with me :-D

THANK YOU!
 

XueSheng

Active Member
Feb 18, 2014
45
0
26
I didn't figure out how to send private messages... so here's the link* to latest git kernel (3.10) with pcie_acs_override patch applied. Patch can be found here. Download is valid until tomorrow. Password is: proxmox

* Update:
Link was removed because it's not valid anymore.
 
Last edited:

mcflym

Active Member
Jul 10, 2013
194
8
38
thank you so much!

Works perfectly!

Each device has his own group now.
 
Last edited:

maxprox

Well-Known Member
Aug 23, 2011
404
34
48
Germany - Nordhessen
www.fair-comp.de
Latest testing kernel 3.10 works great for me.

1. I've enabled allow_unsafe_int
.
and so on.....

Hello XueSheng

Thank you,
and yes, it works for me too (very happy)

the path to the modprobe-file is

Code:
# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

with ".d"


I use also machine model q35 without pcie flag.
Code:
machine: q35
hostpci0: 09:00.0,driver=vfio

The remaining settings like mentioned in the wiki:

https://pve.proxmox.com/wiki/Pci_passthrough

Many thanks!
maxprox
 

XueSheng

Active Member
Feb 18, 2014
45
0
26
the path to the modprobe-file is

Code:
# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

with ".d"
Thanks for this hint! That happens if you don't copy&paste commandlines ;).

Thanks for the reports, I have updated the wiki for

options vfio_iommu_type1 allow_unsafe_interrupts=1
Just saw that the wiki also contains the wrong path to "modprobe.d".
@Spirit: Would you mind correcting this?

As a general note to "pcie_acs_override". I was wondering why I needed this patch with the earlier 3.10 kernels, but now I don't need it anymore. This is because the rhel kernel was patched with quirks for some devices (see LKML as reference).
 

maxprox

Well-Known Member
Aug 23, 2011
404
34
48
Germany - Nordhessen
www.fair-comp.de
Thanks for the reports, I have updated the wiki for

options vfio_iommu_type1 allow_unsafe_interrupts=1

Hello,

today, after the last Updates:
Code:
Will install 2 packages, and remove 0 packages.
===============================================================================
[UPGRADE] pve-kernel-3.10.0-4-pve:amd64 3.10.0-14 -> 3.10.0-15
[UPGRADE] pve-manager:amd64 3.2-22 -> 3.2-23
===============================================================================
My PCIe TV work only with this two lines in the vmid.conf
Code:
hostpci0: 02:00.0,pcie=1,driver=vfio
machine: q35

This grub boot line in /etc/default/grub
Code:
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
...

And WITHOUT this file:
/etc/modprobe.d/iommu_unsafe_interrupts.conf
with this line
Code:
options vfio_iommu_type1 allow_unsafe_interrupts=1

With this file and its option I get the following error:
Code:
kvm: -device vfio-pci,host=02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio: Error: Failed to setup INTx fd: Device or resource busy
kvm: -device vfio-pci,host=02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: Device initialization failed.
kvm: -device vfio-pci,host=02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: Device 'vfio-pci' could not be initialized
start failed:
...

Without the iommu and interrupt option it works fine!

Thank you and regards
maxprox
 

kobuki

Well-Known Member
Dec 30, 2008
470
25
48
I'm running into similar problems with my Intel DQ67SW-based home server runing PVE. The problems started with upgrading from 3.2 to 3.3, kernel went from 2.6.32-30-pve to 2.6.32-32-pve. I'll try to sum up what happened. The card I'm passing through is an IBM Serveraid M1015 in IT mode for ZFS in a VM. Relevant config line is simply: "hostpci0: 01:00.0", nothing else.

1. PVE 3.2, kernel 2.6.32-30-pve, pve-qemu-kvm 1.7-8, kernel parameter intel_iommu=on -- working

2. PVE 3.3, kernel 2.6.32-32-pve, pve-qemu-kvm 2.1-8, kernel parameter intel_iommu=on -- NOT working, similar problem as here: http://forum.proxmox.com/threads/18366-Kernel-2-6-32-28-not-booting-with-iommu-and-pci-passthrough, message endlessly repeating on the console, then server freezes with a kernel panic, see image

3. same as above, but kernel parameter intel_iommu=igfx_off -- NOT working, system can boot and is stable, but same problem as first post in this thread, no IOMMU found

4. same as above, but kernel parameter intel_iommu=on, and pve-qemu-kvm downgraded to 1.7-8 -- working again

Since I need OVZ, I can't upgrade to 3.10 on this server, so I can't use vfio+pcie here. I'd like to use and test the new features in KVM 2.1 though. Any ideas are very welcome.
 

Attachments

  • err.jpg
    err.jpg
    118.2 KB · Views: 17
Last edited:

eric_f

New Member
Oct 14, 2011
13
0
1
Hi,
pcie_acs_override patch has been added to pve-kernel 3.10 in pvetest repository.

Please test :)

Is this in 3.10.0-5? I am using that kernel and still having issues with devices not having their own group.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!