GPU passthrough = failed to set iommu for container

P

Patestismedorgo

New Member
Sep 23, 2021
8
0
1
Hello there,

I followed:
https://pve.proxmox.com/wiki/Pci_passthrough
https://gist.github.com/qubidt/64f617e959725e934992b080e677656f

My info:
PC: HP ProLiant ML310e Gen8
CPU = e3-1220 v2
VGA = Gigabyte HD5450

Configs/outputs:
lspci -v
Code: 
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited / Sapphire Technology Cedar [Radeon HD 5000/6000/7350/8350 Series]
        Physical Slot: 4
        Flags: fast devsel, IRQ 11, IOMMU group 1
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at fbfe0000 (64-bit, non-prefetchable) [disabled] [size=128K]
        I/O ports at 4000 [disabled] [size=256]
        Expansion ROM at fbf00000 [virtual] [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]
        Subsystem: PC Partner Limited / Sapphire Technology Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]
        Physical Slot: 4
        Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 1
        Memory at fbfd0000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

/etc/pve/qemu-server/102.conf
Code: 
#HassOS
agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: IvyBridge,flags=+aes
efidisk0: local-zfs:vm-102-disk-0,size=128K
#VGA
hostpci0: 0000:05:00.0,pcie=1,x-vga=on
#AUDIO
hostpci1: 0000:05:00.1,pcie=1
machine: q35
memory: 6144
name: hassos
net0: virtio=6A:C8:24:C9:F3:F0,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-102-disk-1,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=570420b8-2470-4494-9f87-87d18b2fb505
sockets: 1
startup: order=3,up=120
vmgenid: e0681a25-8365-4a50-bf0d-c3cb51c7ebc1
vmstatestorage: local-zfs

/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=1002:68f9,1002:aa68 disable_vga=1

/etc/modprobe.d/iommu_unsafe_interrupts.conf
Code: 
#empty

/etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=1002:68f9,1002:aa68 vfio-pci.ids=1002:68f9,1002:aa68"

/etc/modprobe.d/blacklist.conf
Code: 
blacklist radeon
blacklist nouveau
blacklist nvidia

dmesg | grep 'remapping'
Code: 
[    0.102799] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.102801] x2apic: IRQ remapping doesn't support X2APIC mode

/etc/modules
Code: 
# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

dmesg | grep -e DMAR -e IOMMU
Code: 
[    0.006664] ACPI: DMAR 0x00000000D5DE4A80 000378 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.006698] ACPI: Reserving DMAR table memory at [mem 0xd5de4a80-0xd5de4df7]
[    0.036506] DMAR: IOMMU enabled
[    0.102489] DMAR: Host address width 39
[    0.102489] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.102493] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
[    0.102495] DMAR: RMRR base: 0x000000d5ffd000 end: 0x000000d5ffffff
[    0.102497] DMAR: RMRR base: 0x000000d5ff6000 end: 0x000000d5ffcfff
[    0.102498] DMAR: RMRR base: 0x000000d5f93000 end: 0x000000d5f94fff
[    0.102499] DMAR: RMRR base: 0x000000d5f8f000 end: 0x000000d5f92fff
[    0.102501] DMAR: RMRR base: 0x000000d5f7f000 end: 0x000000d5f8efff
[    0.102501] DMAR: RMRR base: 0x000000d5f7e000 end: 0x000000d5f7efff
[    0.102502] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[    0.102503] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[    0.102504] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
[    0.102568] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff]
[    0.102570] DMAR: RMRR base: 0x000000d5dee000 end: 0x000000d5deefff
[    0.102571] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.102572] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.102573] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.102574] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.102799] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.905928] DMAR: No ATSR found
[    0.905929] DMAR: dmar0: Using Queued invalidation
[    0.909758] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.677461] DMAR: DRHD: handling fault status reg 2
[    1.677528] DMAR: [INTR-REMAP] Request device [01:00.0] fault index 19 [fault reason 38] Blocked an interrupt request due to source-id verification failure
[   42.058940] vfio-pci 0000:05:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[  214.680913] vfio-pci 0000:05:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[  268.700313] vfio-pci 0000:05:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[  650.088463] vfio-pci 0000:05:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

What am I still missing here?
 
Last edited:
I don't see a question or a description of your problem, but could you please report on your IOMMU groups using: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done?
 
The output of what you asked:
Code: 
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)
        Kernel driver in use: ie31200_edac
        Kernel modules: ie31200_edac
IOMMU group 10 00:1f.0 ISA bridge [0601]: Intel Corporation C204 Chipset LPC Controller [8086:1c54] (rev 05)
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
IOMMU group 10 00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [8086:1c02] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
        Kernel driver in use: ahci
        Kernel modules: ahci
IOMMU group 11 03:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5717 Gigabit Ethernet PCIe [14e4:1655] (rev 10)
        DeviceName: NIC Port 1
        Subsystem: Hewlett-Packard Company NetXtreme BCM5717 Gigabit Ethernet PCIe [103c:18d2]
        Kernel driver in use: tg3
        Kernel modules: tg3
IOMMU group 11 03:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5717 Gigabit Ethernet PCIe [14e4:1655] (rev 10)
        DeviceName: NIC Port 2
        Subsystem: Hewlett-Packard Company NetXtreme BCM5717 Gigabit Ethernet PCIe [103c:18d2]
        Kernel driver in use: tg3
        Kernel modules: tg3
IOMMU group 12 01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 05)
        Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
        Kernel modules: hpwdt
IOMMU group 12 01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200EH [102b:0533]
        Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
        Kernel driver in use: mgag200
        Kernel modules: mgag200
IOMMU group 12 01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 05)
        Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
        Kernel driver in use: hpilo
        Kernel modules: hpilo
IOMMU group 12 01:00.4 USB controller [0c03]: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3300] (rev 02)
        Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci_hcd
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
        Kernel driver in use: pcieport
IOMMU group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
        Kernel driver in use: pcieport
IOMMU group 1 05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
        Subsystem: PC Partner Limited / Sapphire Technology Cedar [Radeon HD 5000/6000/7350/8350 Series] [174b:e164]
        Kernel driver in use: vfio-pci
        Kernel modules: radeon
IOMMU group 1 05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
        Subsystem: PC Partner Limited / Sapphire Technology Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [174b:aa68]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
IOMMU group 2 00:06.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:015d] (rev 09)
        Kernel driver in use: pcieport
IOMMU group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family USB Enhanced Host Controller [103c:330d]
        Kernel driver in use: ehci-pci
        Kernel modules: ehci_pci
IOMMU group 4 00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)
        Kernel driver in use: pcieport
IOMMU group 5 00:1c.4 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [8086:1c18] (rev b5)
        Kernel driver in use: pcieport
IOMMU group 6 00:1c.6 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 [8086:1c1c] (rev b5)
        Kernel driver in use: pcieport
IOMMU group 7 00:1c.7 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 [8086:1c1e] (rev b5)
        Kernel driver in use: pcieport
IOMMU group 8 00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family USB Enhanced Host Controller [103c:330d]
        Kernel driver in use: ehci-pci
        Kernel modules: ehci_pci
IOMMU group 9 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)

I also did this one:
find /sys/kernel/iommu_groups/ -type 1
Code: 
/sys/kernel/iommu_groups/7/devices/0000:00:1c.7
/sys/kernel/iommu_groups/5/devices/0000:00:1c.4
/sys/kernel/iommu_groups/3/devices/0000:00:1a.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.1
/sys/kernel/iommu_groups/1/devices/0000:05:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:05:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.6
/sys/kernel/iommu_groups/4/devices/0000:00:1c.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.4
/sys/kernel/iommu_groups/12/devices/0000:01:00.2
/sys/kernel/iommu_groups/12/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:06.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1e.0

lspci -n -s 05:00
Code: 
05:00.0 0300: 1002:68f9
05:00.1 0403: 1002:aa68

The question, a bit strangely formulated at the end of my post: why does it still not work?
But looking at it now, I see PCI 00:01 is also on the same IOMMU group. I think this is the problem?
 
Last edited:
Bridges are not a problem, so IOMMU group 1 looks fine for passthrough.

Sorry, I did not see this message in your log the first time: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. It means that your system with its current BIOS won't be able to do PCI passthrough. There are several people with the same issue on this forum, maybe one of their fixes or work-arounds work for you?
 
Yes, I saw it too afterwards, my mistake. Searched around and basically HP and this problem are a match in heaven.

Found this article, tried it out but did not make any difference. Then came along this information here, tried it, and now it works.
But what I can't seem to find in a quick glance is too see if it's available now in host. I'm using Home Assistant, know by any chance how I can verify if it exists there?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom