Issues with iommu with 3.10 kernel in pvetest repo

T

tenyosensei

New Member
Jan 9, 2014
2
0
1
Hi,

great work with proxmox! I've installed proxmox 3.1 in a two node cluster fashion with drbd and a quorum iscsi disk and it's wokring very well.

Now I'm triying the pvetest repo, coming from archlinux I really need to play with a more recent qemu and kernel ;-)
I updated a proxmox 3.1 node without issues but I found out that pci passthrough doesn't work, giving a No Iommu found.
I see Dmar and Iommu messages at boot and in grub.cfg is set intel_iommu=on.

"dmesg | grep -e DMAR -e IOMMU" output:
[ 0.000000] ACPI: DMAR 00000000cc3e3920 00080 (v01 INTEL HSW 00000001 INTL 00000001)[ 0.000000] Intel-IOMMU: enabled
[ 0.033050] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.033119] IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
[ 0.787937] DMAR: No ATSR found
[ 0.787967] IOMMU 0 0xfed90000: using Queued invalidation
[ 0.787971] IOMMU: Setting RMRR:
[ 0.787984] IOMMU: Setting identity map for device 0000:00:1d.0 [0xcc28b000 - 0xcc297fff]
[ 0.788022] IOMMU: Setting identity map for device 0000:00:1a.0 [0xcc28b000 - 0xcc297fff]
[ 0.788051] IOMMU: Setting identity map for device 0000:00:14.0 [0xcc28b000 - 0xcc297fff]
[ 0.788073] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.788083] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Message from starting a vm with pci passthrough:
kvm: -device pci-assign,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10: No IOMMU found. Unable to assign device "hostpci0"
kvm: -device pci-assign,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10: Device initialization failed.
kvm: -device pci-assign,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10: Device 'kvm-pci-assign' could not be initialized
TASK ERROR: start failed: command '/usr/bin/kvm -id 201 -chardev 'socket,id=qmp,path=/var/run/qemu-server/201.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/201.vnc,x509,password -pidfile /var/run/qemu-server/201.pid -daemonize -name fwslave -smp 'sockets=1,cores=1' -nodefaults -boot 'menu=on' -vga cirrus -cpu host,+x2apic -k it -m 1024 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'pci-assign,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'pci-assign,host=02:00.1,id=hostpci1,bus=pci.0,addr=0x11' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/proxmox-slave/vm-201-disk-1,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100'' failed: exit code 1

The same vm is working fine on pve3.1 and on an archlinux box with qemu compiled from git and latest kernel, on identical hardware.
I can help with test of new build from the pvetest repo.
Please tell me if this forum is not the right place for talking about the testing branch.

Keep up the good work.

Best regards,
Matteo
 
spirit said:
update : vfio is not yet implemented, only option have been added.

I'll try to have a lot at pci passthrough next week.
Click to expand...

Thank you very much for the info!
I'll downgrade to kernel 2.6.32-27 from pvetest for now.
With this kernel and qemu 1.7 pci passthrough works fine, but i still can't boot nas4free 64bit image as i do on archlinux.
Good luck with pci passtrhough work! I'll be happy to test if needed :)

Best regards,
Matteo
 
I have the same issue with the 3.10 kernel (tried both the pve-kernel-3.10.0-1 and linux-kernel-3.10-0.bpo.3-amd64). Here is a snippet from my dmesg log:

[ 0.019439] dmar: Host address width 39
[ 0.019442] dmar: DRHD base: 0x000000fbffe000 flags: 0x1
[ 0.019452] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.019454] dmar: RMRR base: 0x000000000ec000 end: 0x000000000effff
[ 0.019456] dmar: RMRR base: 0x000000bf7dc000 end: 0x000000bf7dbfff
[ 0.019457] dmar: ATSR flags: 0x0
[ 0.019564] ------------[ cut here ]------------
[ 0.019571] WARNING: at /build/linux-9_tuUh/linux-3.10.11/drivers/iommu/intel_irq_remapping.c:533 intel_irq_remapping_supported+0
x35/0x77()
[ 0.019573] This system BIOS has enabled interrupt remapping
[ 0.019573] on a chipset that contains an erratum making that
[ 0.019573] feature unstable. To maintain system stability
[ 0.019573] interrupt remapping is being disabled. Please
[ 0.019573] contact your BIOS vendor for an update
[ 0.019577] Modules linked in:
[ 0.019580] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10-0.bpo.3-amd64 #1 Debian 3.10.11-1~bpo70+1
[ 0.019582] Hardware name: System manufacturer System Product Name/P6T6 WS REVOLUTION, BIOS 0905 12/24/2010
[ 0.019584] ffffffff8138cac3 0000000000000000 ffffffff8103be31 00000000000080d0
[ 0.019587] ffff8806268f5e38 0000000000000246 0000000000000246 00000000ffffffff
[ 0.019590] 0000000000000000 000000000000b040 ffffffff8103be97 ffffffff81532f6d
[ 0.019593] Call Trace:
[ 0.019598] [<ffffffff8138cac3>] ? dump_stack+0xd/0x17
[ 0.019603] [<ffffffff8103be31>] ? warn_slowpath_common+0x5f/0x77
[ 0.019606] [<ffffffff8103be97>] ? warn_slowpath_fmt_taint+0x3d/0x42
[ 0.019611] [<ffffffff8138ffb3>] ? _raw_spin_lock_irqsave+0x14/0x35
[ 0.019613] [<ffffffff816f3206>] ? intel_irq_remapping_supported+0x35/0x77
[ 0.019618] [<ffffffff816c3bda>] ? enable_IR+0x7/0x38
[ 0.019620] [<ffffffff816c3e7c>] ? enable_IR_x2apic+0x84/0x13c
[ 0.019623] [<ffffffff816c58f0>] ? default_setup_apic_routing+0xd/0x65
[ 0.019626] [<ffffffff816c1ee3>] ? native_smp_prepare_cpus+0x2dd/0x313
[ 0.019630] [<ffffffff816b3de1>] ? kernel_init_freeable+0x85/0x1c5
[ 0.019633] [<ffffffff813762ac>] ? rest_init+0x70/0x70
[ 0.019635] [<ffffffff813762b2>] ? kernel_init+0x6/0xd3
[ 0.019638] [<ffffffff8139537c>] ? ret_from_fork+0x7c/0xb0
[ 0.019640] [<ffffffff813762ac>] ? rest_init+0x70/0x70
[ 0.019646] ---[ end trace a01a16a7981db036 ]---
[ 0.019716] Switched APIC routing to physical flat.
[ 0.020264] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.059935] smpboot: CPU0: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz (fam: 06, model: 1a, stepping: 05)
Click to expand...
 
I'm wondering why IOMMU makes trouble with kernel 3.10. That would be a regression for pci-assign.
But it's good to see some progress on vfio-pci. Now there's another patch for vfio-pci integration that should make it usable (devel mailing list). Not sure if it will make it into proxmox 3.2 (it's still not applied to git).
 
Now I've upgraded to proxmox 3.2 beta and kernel 3.10. I get the same error msg as reported by tenyosensei. In my case I think that this is related to the new kvm module which does not support the previous option "kvm allow_unsafe_assigned_interrupts=1". I'm wondering why this option was removed.

Just for testing I've applied the mentioned patch for vfio-pci. Unfortunately my network dies when starting the vm. When starting the proxmox server, everything is ok (I can ping the server and can access it with ssh). But when starting the vm with vfio-pci, the connection dies. Maybe the networkcard shares an interrupt with the passthrough pci device!?
 
That's what's happening when I start a vm with vfio-pci (network goes down):
/var/log/messages:

Code: 
Feb 19 21:14:37 proxmox kernel: [  203.280664] VFIO - User Level meta-driver version: 0.3
Feb 19 21:14:37 proxmox kernel: [  203.284198] ------------[ cut here ]------------
Feb 19 21:14:37 proxmox kernel: [  203.284210] WARNING: at drivers/pci/pci.c:1424 pci_disable_device+0x90/0xa0()
Feb 19 21:14:37 proxmox kernel: [  203.284212] Device pcieport
Feb 19 21:14:37 proxmox kernel: [  203.284212] disabling already-disabled device
Feb 19 21:14:37 proxmox kernel: [  203.284213] Modules linked in: vfio_iommu_type1 vfio_pci vfio nfsv3 vhost_net tun macvtap macvlan kvm_intel kvm ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl nfs lockd fscache sunrpc cifs dns_resolver fuse stv6110x lnbp21 iTCO_wdt iTCO_vendor_support mxm_wmi snd_pcm snd_page_alloc snd_timer snd stv090x soundcore pcspkr serio_raw i2c_i801 i915 wmi asus_atk0110 lpc_ich mfd_core video i2c_algo_bit ddbridge drm_kms_helper dvb_core drm i2c_core acpi_cpufreq mperf ext4 mbcache jbd2 sg ses enclosure ahci libahci aacraid(O) libata r8169 mii
Feb 19 21:14:37 proxmox kernel: [  203.284262] CPU: 0 PID: 3036 Comm: task UPID:proxm Tainted: G           O--------------   3.10.0-1-pve #1
Feb 19 21:14:37 proxmox kernel: [  203.284264] Hardware name: System manufacturer System Product Name/P7H57D-V EVO, BIOS 1903    09/28/2012
Feb 19 21:14:37 proxmox kernel: [  203.284266]  0000000000000009 ffff8803c9fbdce8 ffffffff8160fe8a ffff8803c9fbdd28
Feb 19 21:14:37 proxmox kernel: [  203.284269]  ffffffff81059e70 ffff8803c9fbdd28 ffff880409262000 ffffffff819b8500
Feb 19 21:14:37 proxmox kernel: [  203.284272]  ffff880409262000 ffffffffffffffed 000000000000000c ffff8803c9fbdd88
Feb 19 21:14:37 proxmox kernel: [  203.284276] Call Trace:
Feb 19 21:14:37 proxmox kernel: [  203.284284]  [<ffffffff8160fe8a>] dump_stack+0x19/0x1b
Feb 19 21:14:37 proxmox kernel: [  203.284290]  [<ffffffff81059e70>] warn_slowpath_common+0x70/0xa0
Feb 19 21:14:37 proxmox kernel: [  203.284294]  [<ffffffff81059f56>] warn_slowpath_fmt+0x46/0x50
Feb 19 21:14:37 proxmox kernel: [  203.284298]  [<ffffffff812e8e00>] pci_disable_device+0x90/0xa0
Feb 19 21:14:37 proxmox kernel: [  203.284302]  [<ffffffff812f811e>] pcie_portdrv_remove+0x1e/0x30
Feb 19 21:14:37 proxmox kernel: [  203.284305]  [<ffffffff812ec036>] pci_device_remove+0x46/0xc0
Feb 19 21:14:37 proxmox kernel: [  203.284311]  [<ffffffff813b215f>] __device_release_driver+0x7f/0xf0
Feb 19 21:14:37 proxmox kernel: [  203.284314]  [<ffffffff813b252c>] device_release_driver+0x2c/0x40
Feb 19 21:14:37 proxmox kernel: [  203.284328]  [<ffffffff813b1211>] driver_unbind+0xa1/0xc0
Feb 19 21:14:37 proxmox kernel: [  203.284342]  [<ffffffff813b0444>] drv_attr_store+0x24/0x40
Feb 19 21:14:37 proxmox kernel: [  203.284355]  [<ffffffff812161c0>] sysfs_write_file+0xd0/0x150
Feb 19 21:14:37 proxmox kernel: [  203.284366]  [<ffffffff811a3aa5>] vfs_write+0xc5/0x1f0
Feb 19 21:14:37 proxmox kernel: [  203.284369]  [<ffffffff811a3f92>] SyS_write+0x52/0xa0
Feb 19 21:14:37 proxmox kernel: [  203.284374]  [<ffffffff816200d9>] system_call_fastpath+0x16/0x1b
Feb 19 21:14:37 proxmox kernel: [  203.284376] ---[ end trace a55ea82636e6043a ]---
Feb 19 21:14:37 proxmox kernel: [  203.299170] vmbr0: port 1(eth0) entered disabled state
Feb 19 21:14:37 proxmox kernel: [  203.299301] device eth0 left promiscuous mode
Feb 19 21:14:37 proxmox kernel: [  203.299306] vmbr0: port 1(eth0) entered disabled state
Feb 19 21:14:37 proxmox kernel: [  203.329042] xhci_hcd 0000:07:00.0: remove, state 4
Feb 19 21:14:37 proxmox kernel: [  203.329055] usb usb4: USB disconnect, device number 1
Feb 19 21:14:37 proxmox kernel: [  203.344716] xhci_hcd 0000:07:00.0: USB bus 4 deregistered
Feb 19 21:14:37 proxmox kernel: [  203.344847] xhci_hcd 0000:07:00.0: remove, state 4
Feb 19 21:14:37 proxmox kernel: [  203.344856] usb usb3: USB disconnect, device number 1
Feb 19 21:14:37 proxmox kernel: [  203.345126] xhci_hcd 0000:07:00.0: USB bus 3 deregistered
Feb 19 21:14:38 proxmox kernel: [  203.728752] device tap120i0 entered promiscuous mode
Feb 19 21:14:38 proxmox kernel: [  203.736471] vmbr0: port 1(tap120i0) entered forwarding state
Feb 19 21:14:38 proxmox kernel: [  203.736480] vmbr0: port 1(tap120i0) entered forwarding state
Feb 19 21:14:40 proxmox kernel: [  205.711556] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:40 proxmox kernel: [  205.942539] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:40 proxmox kernel: [  206.096549] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:40 proxmox kernel: [  206.247552] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:46 proxmox kernel: [  212.046505] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:47 proxmox kernel: [  212.792519] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:47 proxmox kernel: [  212.943513] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:47 proxmox kernel: [  213.369377] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:48 proxmox kernel: [  213.941379] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci
Feb 19 21:14:48 proxmox kernel: [  214.206533] usb 2-1.2: reset full-speed USB device number 4 using ehci-pci

This means that in my case neither pci-assign, nor vfio-pci are working with kernel 3.10 ;(.
Any hint how I could get vfio-pci working?
 
Now I've tried a step by step procedure for vfio-pci.

Code: 
# lspci
00:00.0 Host bridge: Intel Corporation Core Processor DRAM Controller (rev 18)
00:01.0 PCI bridge: Intel Corporation Core Processor PCI Express x16 Root Port (rev 18)
00:02.0 VGA compatible controller: Intel Corporation Core Processor Integrated Graphics Controller (rev 18)
00:16.0 Communication controller: Intel Corporation 5 Series/3400 Series Chipset HECI Controller (rev 06)
00:1a.0 USB controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 06)
00:1c.0 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 1 (rev 06)
00:1c.4 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 5 (rev 06)
00:1c.5 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 6 (rev 06)
00:1c.7 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 8 (rev 06)
00:1d.0 USB controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation 5 Series Chipset LPC Interface Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation 5 Series/3400 Series Chipset SMBus Controller (rev 06)
01:00.0 RAID bus controller: Adaptec Series 7 6G SAS/PCIe 3 (rev 01)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
05:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
[B]09:00.0 Multimedia controller: Digital Devices GmbH Octopus LE DVB adapter[/B]
3f:00.0 Host bridge: Intel Corporation Core Processor QuickPath Architecture Generic Non-core Registers (rev 05)
3f:00.1 Host bridge: Intel Corporation Core Processor QuickPath Architecture System Address Decoder (rev 05)
3f:02.0 Host bridge: Intel Corporation Core Processor QPI Link 0 (rev 05)
3f:02.1 Host bridge: Intel Corporation Core Processor QPI Physical 0 (rev 05)
3f:02.2 Host bridge: Intel Corporation Core Processor Reserved (rev 05)
3f:02.3 Host bridge: Intel Corporation Core Processor Reserved (rev 05)

09:00.0 is the device which I want to passthrough.

Code: 
# readlink /sys/bus/pci/devices/0000\:09\:00.0/iommu_group
../../../../../../kernel/iommu_groups/5

This device is part of iommu group 5.

Code: 
# lspci -n -s 0000:09:00.0
09:00.0 0480: dd01:0003
# echo 0000:09:00.0 > /sys/bus/pci/devices/0000\:09\:00.0/driver/unbind
# echo dd01 0003 > /sys/bus/pci/drivers/vfio-pci/new_id

Code: 
# ls -l /sys/bus/pci/devices/0000\:09\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:00:1c.4 -> ../../../../devices/pci0000:00/0000:00:1c.4
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:00:1c.5 -> ../../../../devices/pci0000:00/0000:00:1c.5
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:02:00.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:06:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:01.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:06:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:05.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:07.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:06:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:09.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:01.0/0000:07:00.0
lrwxrwxrwx 1 root root 0 Feb 20 14:54 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:06:07.0/0000:09:00.0

Thus nearly all pci devices are in the same iommu group... 02:00.0 is my network adapter. When I try to unbind this device, network goes down (as shown in the previous posts).

Anything I can do at this point to get 09:00.0 into a different (smaller) iommu group?
 
XueSheng said:
Anything I can do at this point to get 09:00.0 into a different (smaller) iommu group?
Click to expand...
I know, this is kind of a monologue, but maybe this is helpful if someone else stumbles over the same issue.

While doing some research I found a patch which adds a quirk in regards to PCIe ACS (kernel mailing list). This patch actually does resolve my issue, that all pci devices are in only one iommu group (which makes it impossible to pass a single device to a vm). This quirk must be enables/configured with boot parameters (e.g. pcie_acs_override=downstream):

Code: 
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1-pve root=/dev/mapper/pve-root ro [B]pcie_acs_override=downstream[/B] quiet intel_iommu=on

Additionally I've enabled unsafe interrupts (I didn't test if this is really needed after applying the acs_override patch... at least before applying the patch I had to enable unsafe interrupts):
Code: 
# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Of course vfio-pci patch must be applied to QemuServer.pm and vm conf must be customized. At least I'm now able to passthrough my PCIe device with vfio-pci. Further testing will show if there are any issues left.

Would be great to get some feedback if others have issues with kernel 3.10 and iommu too.
 
Last edited:
I'm not familiar with the state of passthrough support in Proxmox 3.2 using Kernel 3.10. I want to pass through a SATA controller - can someone who has already done this write a quick tutorial of the needed commands? As far as I know the wiki is not up to date: https://pve.proxmox.com/wiki/Pci_passthrough
 
ringerl said:
I'm not familiar with the state of passthrough support in Proxmox 3.2 using Kernel 3.10. I want to pass through a SATA controller - can someone who has already done this write a quick tutorial of the needed commands? As far as I know the wiki is not up to date: https://pve.proxmox.com/wiki/Pci_passthrough
Click to expand...
Proxmox 3.2 didn't introduce any changes to pci passthrough (thus the wiki is still valid). You should also use the search function of this forum if you need further input on how to passthrough pci devices. I would recommend to start testing with stock kernel first. If this works, you can start testing with kernel 3.10.
 
XueSheng said:
I know, this is kind of a monologue, but maybe this is helpful if someone else stumbles over the same issue.

While doing some research I found a patch which adds a quirk in regards to PCIe ACS (kernel mailing list). This patch actually does resolve my issue, that all pci devices are in only one iommu group (which makes it impossible to pass a single device to a vm). This quirk must be enables/configured with boot parameters (e.g. pcie_acs_override=downstream):

Code: 
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1-pve root=/dev/mapper/pve-root ro [B]pcie_acs_override=downstream[/B] quiet intel_iommu=on

Additionally I've enabled unsafe interrupts (I didn't test if this is really needed after applying the acs_override patch... at least before applying the patch I had to enable unsafe interrupts):
Code: 
# cat /etc/modprobe/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Of course vfio-pci patch must be applied to QemuServer.pm and vm conf must be customized. At least I'm now able to passthrough my PCIe device with vfio-pci. Further testing will show if there are any issues left.

Would be great to get some feedback if others have issues with kernel 3.10 and iommu too.
Click to expand...

Hi,

what i have to do to enable and use vfio-pci? I'm stuck on passthrough of my Video-Card (Code43 in Windows VM). The device is visible and i can install drivers without issues but it doesn't work...

Thank you!
 
XueSheng said:
Proxmox 3.2 didn't introduce any changes to pci passthrough (thus the wiki is still valid). You should also use the search function of this forum if you need further input on how to passthrough pci devices. I would recommend to start testing with stock kernel first. If this works, you can start testing with kernel 3.10.
Click to expand...

Kernel 3.10 is what I would need - this is why I wrote "Proxmox 3.2 using kernel 3.10". I believe I would need vfio (at least I see references here http://forum.proxmox.com/threads/17690-Beta-packages-for-Proxmox-VE-3-2-pvetest/page2)
 
Just a short update to my comment on using "pcie_acs_override" patch.
After upgrading to latest git kernel pve-kernel-3.10.0-3-pve_3.10.0-11_amd64 I don't need pcie_acs_override anymore. My pci devices are now split into separate iommu_groups without any hickups.
 
XueSheng said:
Just a short update to my comment on using "pcie_acs_override" patch.
After upgrading to latest git kernel pve-kernel-3.10.0-3-pve_3.10.0-11_amd64 I don't need pcie_acs_override anymore. My pci devices are now split into separate iommu_groups without any hickups.
Click to expand...

Hello XueSheng,

I'm working with iommu and pci-passthrough for my virtual video recorder (http://www.easy-vdr.de/forum/index.php?topic=16254.msg146563#msg146563)
The last Kernel that works for me was version 2.6.32-27, have a look at here: http://188.165.151.221/threads/18366-Kernel-2-6-32-28-not-booting-with-iommu-and-pci-passthrough
If you think the last testing Kernel is working again with IOMMU (?) You can then briefly explain how you solved the setup for your pci-passthrough?

thanks for your help,
maxprox
 
Last edited:
maxprox said:
Hello XueSheng,
If you think the last testing Kernel is working again with IOMMU (?) You can then briefly explain how you solved the setup for your pci-passthrough?
Click to expand...

Latest testing kernel 3.10 works great for me.

1. I've enabled allow_unsafe_interrupts, otherwise pci-passthrough didn't work.
Code: 
# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

2. Add vfio config to your VMID.conf
Code: 
hostpci0: 09:00.0,driver=vfio

Alternatively you can also try machine model q35 and pcie flag.
Code: 
machine: q35
hostpci0: 09:00.0,driver=vfio,pcie=1

I'm using this config for a couple of weeks now, without any hickups.
 
Last edited:
how can i install this kernel?

apt-get won't work?

I have the same issues with th iommu groups...

thanks!

Edit:

I downloaded pve-kernel-3.10.0-4-pve_3.10.0-14_amd64.deb , will have a look :)

Edit 2:
The devices are still in the same group :/
 
Last edited:
yes i think i will try this, but i wanna wait what spirit is thinking about my issue wirh my Radeon-Card.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back