Help with PCI Express passthrough (ACS - IOMMU issue) + kernel bugfix

[Jeky]

Hello everyone, I'm new in this form and with Proxmox.
<little presentation on me, you can skip this>
I've got some experience with Linux and virtualization (mainly with VMWare where SSH is the first thing I enable).
I want to expertise with Proxmox because I'm a consultant, and often VMWare license are too expensive for some customers, and also free ESXi is too limitated.
</little presentation on me, you can skip this>

Since I just want to "play" with a new game, I've installed Proxmox on what will be my new home NAS.
- Asrock J4205-ITX Mainboard with 16GB RAM
- Fractal design Node 304 Case
- 2x WD Red 4TB, 1x250GB Samsung EVO SSD

The problem: I want to passthrough one of the two SATA controller to a VM.
The CPU is VT-d capable and I've already made it working with VMWare, just few clicks:


I want to pass the ASMedia controller, since it is an "added" controller, not the native Intel CPU one (that I want to keep for the SSD).
I've enabled "intel_iommu=on" but my IOMMU groups are like that:

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge [8086:5af0] (rev 0b)
IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Integrated Graphics Controller [8086:5a84] (rev 0b)
IOMMU Group 2 00:0f.0 Communication controller [0780]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine [8086:5a9a] (rev 0b)
IOMMU Group 3 00:12.0 SATA controller [0106]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller [8086:5ae3] (rev 0b)
IOMMU Group 4 00:13.0 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 [8086:5ad8] (rev fb)
IOMMU Group 4 00:13.1 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 [8086:5ad9] (rev fb)
IOMMU Group 4 00:13.2 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 [8086:5ada] (rev fb)
IOMMU Group 4 00:13.3 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 [8086:5adb] (rev fb)
IOMMU Group 4 01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11)
IOMMU Group 4 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 5 00:15.0 USB controller [0c03]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI [8086:5aa8] (rev 0b)
IOMMU Group 6 00:1f.0 ISA bridge [0601]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface [8086:5ae8] (rev 0b)
IOMMU Group 6 00:1f.1 SMBus [0c05]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller [8086:5ad4] (rev 0b)

Group 4 is a BIG group and both ASMedia controller and Realteck Ethernet Controller belongs to this Group!

When I try to pass 03:00:0 SATA Controller, my network card stop to work on my Proxmox host

Searching around I've fonud that passing a device within a big IOMMU Group, may impact other devices, and this is the case. I've also found that Group splitting is something related to ACS and that a patch is required on kernel. Proxmox already apply this patch (but there is a BUG, I'll tell you later).

I've tried using "pcie_acs_override=downstream" and also "pcie_acs_override=multifunction" but it don't make any change.

With lspci -t I've noticed that my devices are not "root-connected":

-[0000:00]-+-00.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge
           +-02.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Integrated Graphics Controller
           +-0f.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine
           +-12.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller
           +-13.0-[01]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           +-13.1-[02]--
           +-13.2-[03]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           +-13.3-[04]--
           +-15.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI
           +-1f.0  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface
           \-1f.1  Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller

Here lspci

00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Integrated Graphics Controller (rev 0b)
00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
00:13.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 (rev fb)
00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

Here lspci -n

00:00.0 0600: 8086:5af0 (rev 0b)
00:02.0 0300: 8086:5a84 (rev 0b)
00:0f.0 0780: 8086:5a9a (rev 0b)
00:12.0 0106: 8086:5ae3 (rev 0b)
00:13.0 0604: 8086:5ad8 (rev fb)
00:13.1 0604: 8086:5ad9 (rev fb)
00:13.2 0604: 8086:5ada (rev fb)
00:13.3 0604: 8086:5adb (rev fb)
00:15.0 0c03: 8086:5aa8 (rev 0b)
00:1f.0 0601: 8086:5ae8 (rev 0b)
00:1f.1 0c05: 8086:5ad4 (rev 0b)
01:00.0 0200: 10ec:8168 (rev 11)
03:00.0 0106: 1b21:0612 (rev 02)

So I wanted to try to specify my device (and/or its parent device) using "pcie_acs_override=id:nnnn:nnnn" and discovered a BUG!

When I specify the device id I always get "PCIe ACS invalid ID" in dmesg!
This because the patch has got an error:
hxxps://git.proxmox.com/?p=pve-kernel.git;a=blob;f=override_for_missing_acs_capabilities.patch;h=7052f5a040de6e80aa4746e2caf1d4e97371604e;hb=54a9e5a210ac9494b27249c96ec07c212ad3106d
On line 123,124 you can see that the ":" is searched inside the string and than, if ";" not found, fail.
Of course the ";" on line 124 should be ":" as I've found in others version of the same patch:
hxxps://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb

SO, I've used this (hxxps://forum.proxmox.com/threads/tutorial-compile-proxmox-ve-5-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/#post-179193) tutorial and I've recompiled my kernel with the fixed patch.
I no longer receive "invalid ID" error, but the situation is still the same

Some other stuff may help: dmesg|grep -e DMAR -e IOMMU

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] ACPI: DMAR 0x000000006D9BB4D0 0000A8 (v01 INTEL  EDK2     00000003 BRXT 0100000D)
[    0.000000] DMAR: IOMMU enabled
[    0.068770] DMAR: Host address width 39
[    0.068773] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
[    0.068790] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.068792] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
[    0.068803] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.068806] DMAR: RMRR base: 0x0000006d592000 end: 0x0000006d5b1fff
[    0.068808] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007fffffff
[    0.068812] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
[    0.068813] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
[    0.068815] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.070815] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.224690] DMAR: No ATSR found
[    1.225561] DMAR: dmar0: Using Queued invalidation
[    1.225569] DMAR: dmar1: Using Queued invalidation
[    1.225663] DMAR: Setting RMRR:
[    1.225853] DMAR: Setting identity map for device 0000:00:02.0 [0x6f800000 - 0x7fffffff]
[    1.225916] DMAR: Setting identity map for device 0000:00:15.0 [0x6d592000 - 0x6d5b1fff]
[    1.225927] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.225978] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.226060] DMAR: Intel(R) Virtualization Technology for Directed I/O

dmesg | grep -i acs (this is just one of the various try)

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=id:8086:5ad8,id:8086:5ad9,id:8086:5ada,id:8086:5adb,id:10ec:8168,id:1b21:0612
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] ACPI: FACS 0x000000006D9C8080 000040
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=id:8086:5ad8,id:8086:5ad9,id:8086:5ada,id:8086:5adb,id:10ec:8168,id:1b21:0612

As you may see I've tried to manage it myself, but it seems I need some help!

If you're curious I want to pass my ASMedia controller since my desidered configuration will have an additional SATA controller on PCI-E x1 port of the mainboard and:
Proxmox host with Intel SATA controller and one SSD
-> VM1 with one ASMedia controller + 2xWD Red 4TB - OpenMediaVault for Sharing my Movies
-> VM2 with the other ASMedia controller + 2xWD Red 1TB - FreeNAS with ZFS and dedup

I'll also try to use ZFS with dedup on Proxmox, I know it is possible... FreeNAS is just another "new" (for me) thing I'd like to try

Any help/idea is well accepted.

Remember, it is not an hardware limitation since I've got it working with VMware !!!
Also, I've successfully passed the Intel 00:12:0 controller to a VM, but it is not what I want.
Also because I want to pass two controllers to two VM. (I don't had the additional controller during "screenshots" but I've already installed it, and it was 04:00:0 just behind the other ASMedia, with the same problem in Proxmox but not in VMware)

\Jeky

P.S.
I've also tried with "vfio_iommu_type1 allow_unsafe_interrupts=1" and playing with vfio / pci_stub .. the problem is the same, if I start the VM with the ASMedia controller passthrough, the network card stop working.

pveperf
CPU BOGOMIPS:      11980.80
REGEX/SECOND:      960974
HD SIZE:           56.84 GB (/dev/mapper/pve-root)
BUFFERED READS:    508.26 MB/sec
AVERAGE SEEK TIME: 0.10 ms
FSYNCS/SECOND:     550.46
DNS EXT:           39.89 ms
DNS INT:           17.63 ms (lan)

 

[fabian]

thanks for catching the typo in the acs override patch, queued for the next kernel update:
https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=cfe8ba0b247c813c7636995f0cbfada3f798d0ed

 

[Jeky]

No way? Should I try to ask to the mainboard vendor?
Thanks
\Jeky

 

[dcsapak]

if the controller and network card are in the same iommu group, they may interfere with one another even with the acs patch, so if it does not work, there is not much from our side to do

 

[Jeky]

Thank you for your message.
I think it is strange that so many items are on the same iommu group even with the acs patch, isn't it?
I think it is also strange that VMware allowed me to passthrough the controller without problems...

What is your advice in my situation, ask on asrock forum may be a good idea?

Thank you again
\Jeky

 

[dcsapak]

I think it is strange that so many items are on the same iommu group even with the acs patch, isn't it?

did you try with pcie_acs_override=downstream instead of using the specific ids ?

 

[Jeky]

Yes, it is on the initial post:
<<I've tried using "pcie_acs_override=downstream" and also "pcie_acs_override=multifunction" but it don't make any change.>>

 

[DeMoB]

I've discovered I have a very similar issue to yours with an Asrock J3455B-ITX board, in that among other devices the onboard NIC is in the same group as the PCIe slot, so can't pass through either to a VM without the host losing access to the other.

I've not had chance to try the following config myself yet, but have you tried the following options all together?

In Grub:
"pcie_acs_override=downstream,multifunction"

In /etc/modprobe.d/iommu_unsafe_interrupts.conf:
"options vfio_iommu_type1 allow_unsafe_interrupts=1"

In /etc/modprobe.d/vfio.conf add the Ethernet and SATA (but not the PCI bridges) :
options vfio-pci ids=[10ec:8168],[1b21:0612]

I've left the PCI bridges out due to a note on the ArchLinux Wiki (hxxps://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Gotchas) as their PCI passthrough documentation is a lot more detailed, and I figure it's worth a try.