[SOLVED] Is ACS_Override Needed For My PCIe-Passthrough Setup?

M

mhayhurst

Renowned Member
Jul 21, 2016
111
7
83
43
Hello,

I am testing PCIe Passthrough on a friend's Proxmox machine (kernel:
4.4.44-1-pve) before I purchase any hardware for myself so I wanted to ask before moving forward with patching the kernel if that becomes necessary. I'm currently trying to initiate GPU PCIe Passthrough to a Windows VM and it appears IOMMU grouping is working correctly but I wanted to make sure. Here is a list of my PCI devices:

Code: 
root@proxmox3:~# lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1c.7 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
04:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 20)
05:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 13)
06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
07:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
08:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
08:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
08:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
08:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
0c:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0d:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

IOMMU appears to be initialized correctly:

Code: 
root@proxmox3:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000CE522760 000080 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.029111] DMAR: Host address width 39
[    0.029112] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.029118] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.029119] DMAR: RMRR base: 0x000000cee95000 end: 0x000000ceea4fff
[    0.029120] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.029121] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.029122] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.029122] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.029265] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.501133] DMAR: No ATSR found
[    0.501185] DMAR: dmar0: Using Queued invalidation
[    0.501194] DMAR: Setting RMRR:
[    0.501208] DMAR: Setting identity map for device 0000:00:14.0 [0xcee95000 - 0xceea4fff]
[    0.501226] DMAR: Setting identity map for device 0000:00:1a.0 [0xcee95000 - 0xceea4fff]
[    0.501243] DMAR: Setting identity map for device 0000:00:1d.0 [0xcee95000 - 0xceea4fff]
[    0.501255] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.501260] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.501269] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    7.343090] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    7.343091] AMD IOMMUv2 functionality not available on this system

And here is my IOMMU grouping:

Code: 
root@proxmox3:~# ./iommu_groups 

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)

IOMMU Group 10 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0)

IOMMU Group 11 00:1c.7 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 [8086:8c9e] (rev d0)

IOMMU Group 12 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]

IOMMU Group 13 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4]

IOMMU Group 13 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]

IOMMU Group 13 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]

IOMMU Group 14 04:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 20)

IOMMU Group 15 05:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller [1969:e091] (rev 13)

IOMMU Group 16 06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]

IOMMU Group 17 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]

IOMMU Group 18 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]

IOMMU Group 19 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)

IOMMU Group 1 00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller [8086:0c09] (rev 06)

IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]

IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]

IOMMU Group 1 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]

IOMMU Group 1 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]

IOMMU Group 20 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]

IOMMU Group 21 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]

IOMMU Group 21 0c:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]

IOMMU Group 22 0d:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)

IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]

IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]

IOMMU Group 4 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]

IOMMU Group 5 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]

IOMMU Group 6 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)

IOMMU Group 7 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0)

IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)

IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)

I have two graphics cards (and their corresponding audio devices) which are: 01:00.0; 01:00.1; 02:00.0; 02:00.1 and have been allocated to IOMMU Group 1. However, you will notice that the PCI Express Controllers (00:01.0 and 00:01.2) have also been allocated to IOMMU Group 1. If I'm reading this article correctly: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF under the section called: "Gotchas", this is acceptable so long as those PCI Express Controllers contain everything that I want to passthrough to the VM, am I correct in my thinking? If that is correct, then theoretically I should be able to passthrough my two graphics cards (with the audio devices) to my Windows VM without patching my kernel with the ACS_Override patch? I also wanted to mention a few caveats that I thought were relevant to this situation:

  1. My processor is an: Intel i7-4770 with integrated graphics (which for some reason is not showing up in lspci), but my main concern is will the integrated graphics create problems when trying to pass my graphics cards to the VM?
  2. The graphics cards I am using are older and do not have UEFI boot roms available but my mainboard uses UEFI. According to the Proxmox Wiki, this does not appear to be an issue but I wanted to make sure?
  3. If down the road I wanted to passthrough one of the two graphics cards to another VM, will I need to move it to another PCIe slot and see if that changes the IOMMU grouping, purchase a graphics card that is not identical to the other one and/or patch the kernel with the ACS_Override patch?
I would be happy to provide additional information if necessary and thanks for the help!
 
mhayhurst said:
If that is correct, then theoretically I should be able to passthrough my two graphics cards (with the audio devices) to my Windows VM without patching my kernel with the ACS_Override patch? I also wanted to mention a few caveats that I thought were relevant to this situation:
Click to expand...
this *shoud* work if you pass either both gpus to a single vm, or a single gpu to a single vm and the other is not active anywhere
what i guess will not work is to pass them through to two different vms

mhayhurst said:
My processor is an: Intel i7-4770 with integrated graphics (which for some reason is not showing up in lspci), but my main concern is will the integrated graphics create problems when trying to pass my graphics cards to the VM?
Click to expand...
this should make no problem (i have a similar setup here), but as to it not showing up, maybe you have it disabled in your motherboard? it is necessary that the host has also a gpu, else it will try to use one of those you want to pass through (or you disable graphics output altogether, but this is more involved)

mhayhurst said:
The graphics cards I am using are older and do not have UEFI boot roms available but my mainboard uses UEFI. According to the Proxmox Wiki, this does not appear to be an issue but I wanted to make sure?
Click to expand...
i have here a r7 250E here, also without uefi bios, works without a problem

mhayhurst said:
If down the road I wanted to passthrough one of the two graphics cards to another VM, will I need to move it to another PCIe slot and see if that changes the IOMMU grouping, purchase a graphics card that is not identical to the other one and/or patch the kernel with the ACS_Override patch?
Click to expand...
yes either another pcie slot in another iommu group, or the acs override patch but i do not think it is enough to have a different card
 
dcsapak said:
this *shoud* work if you pass either both gpus to a single vm, or a single gpu to a single vm and the other is not active anywhere
what i guess will not work is to pass them through to two different vms


this should make no problem (i have a similar setup here), but as to it not showing up, maybe you have it disabled in your motherboard? it is necessary that the host has also a gpu, else it will try to use one of those you want to pass through (or you disable graphics output altogether, but this is more involved)


i have here a r7 250E here, also without uefi bios, works without a problem


yes either another pcie slot in another iommu group, or the acs override patch but i do not think it is enough to have a different card
Click to expand...

Hey, thanks for the information! It's nice to speak to someone that actually has this working! I'm out of town right now but will try this when I get back. Regarding hardware, it's difficult to find updated wiki's and/or forums that list newer hardware as being compatible with this setup but as a general rule of thumb, do most high end Intel consumer and high end mainboards support doing this nowadays? I was looking at getting an Intel i7-7700K Kaby Lake Quad-Core and maybe a high-end Asus or MSI mainboard such as: Z270 or Z170 series. I will definitely get graphics cards that support UEFI booting too.
 
i cannot really say how good supported those features are, but i have following setup here:

Mainboard: ASUS Z170-A
CPU: i7-6700k
GPUs: Intel iGPU for the host, Sapphire R7 250E for a Windows VM

my config is this:

Code: 
boot: cdn
bootdisk: scsi0
cores: 4
cpu: host
hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
hotplug: disk,network,usb
ide2: none,media=cdrom
machine: q35
memory: 4096
name: win7pci
net0: virtio=DE:0A:69:CE:46:F8,bridge=vmbr0
numa: 0
onboot: 1
ostype: win7
scsi0: local:3000/vm-3000-disk-1.raw,cache=unsafe,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=f27f45cb-90d5-4d61-b156-ba6aa1af01d0
sockets: 1

I am currently on PVE5 beta, so I cannot really say if it works with PVE4
 
dcsapak said:
i cannot really say how good supported those features are, but i have following setup here:

Mainboard: ASUS Z170-A
CPU: i7-6700k
GPUs: Intel iGPU for the host, Sapphire R7 250E for a Windows VM

my config is this:

Code: 
boot: cdn
bootdisk: scsi0
cores: 4
cpu: host
hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
hotplug: disk,network,usb
ide2: none,media=cdrom
machine: q35
memory: 4096
name: win7pci
net0: virtio=DE:0A:69:CE:46:F8,bridge=vmbr0
numa: 0
onboot: 1
ostype: win7
scsi0: local:3000/vm-3000-disk-1.raw,cache=unsafe,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=f27f45cb-90d5-4d61-b156-ba6aa1af01d0
sockets: 1

I am currently on PVE5 beta, so I cannot really say if it works with PVE4
Click to expand...

Great news! I was able to successfully passthrough my Sapphire Radeon HD 7970 to a Windows 7 VM on the latest Proxmox 4.4-13 (kernel:
4.4.62-1-pve). So thank you for the help! I'm currently spec'ing out a system using the Z270 chipset on the mainboard, an Intel i7-7700K Kaby Lake Quad-Core and maybe a Radeon R9 (nothing too crazy). It will probably be sometime before I can get this built but I will post my results.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back