GPU pass through on PVE 7.2-3

J

jeff feng

New Member
Oct 8, 2022
8
0
1
Hi all:
I got a problem when I want to set GPU pass through to Synology DSM.
I intalled the PVE 7.2-3, I thought the IOMMU was on by default,，when I set new VM on web, I can add network, SATA controller, and it does work in my router and DSM VM system, but when I add GPU to Synology DSM, to transcode in ffmpeg, got only green screen。

CPU info E3-1265L V3 with GPU
and I didn't add any thing to GRUB, VIFO,and pve-blacklist

confused why network adpter and SATA controller working ,but not GPU,thanks for help

check the IOMMU with dmesg | grep -e -DMAR -e IOMMU , seems return error
[ 0.006995] ACPI: DMAR 0x00000000D9915598 0000B8 (v01 INTEL HSW 0000000 1 INTL 00000001)
[ 0.007021] ACPI: Reserving DMAR table memory at [mem 0xd9915598-0xd991564f]
[ 0.117774] DMAR: Host address width 39
[ 0.117775] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.117780] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ec ap f0101a
[ 0.117782] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.117785] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ec ap f010da
[ 0.117787] DMAR: RMRR base: 0x000000d988c000 end: 0x000000d9898fff
[ 0.117789] DMAR: RMRR base: 0x000000dc000000 end: 0x000000de1fffff
[ 0.117791] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
[ 0.117793] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.117794] DMAR-IR: Queued invalidation will be enabled to support x2apic an d Intr-remapping.
[ 0.118360] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.401939] DMAR: No ATSR found
[ 0.401940] DMAR: No SATC found
[ 0.401941] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.401942] DMAR: IOMMU feature sc_support inconsistent
[ 0.401943] DMAR: IOMMU feature pass_through inconsistent
[ 0.401944] DMAR: dmar0: Using Queued invalidation
[ 0.401951] DMAR: dmar1: Using Queued invalidation
[ 0.499147] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 7.841283] i915 0000:00:02.0: [drm] DMAR active, disabling use of stolen mem ory
[ 82.319200] DMAR: DRHD: handling fault status reg 3
[ 82.319207] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc200000 [fault reason 0x05] PTE Write access is not set
[ 82.319212] DMAR: DRHD: handling fault status reg 3
[ 82.319214] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc200000 [fault reason 0x05] PTE Write access is not set
[ 82.319229] DMAR: DRHD: handling fault status reg 3
[ 82.319231] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc221000 [fault reason 0x05] PTE Write access is not set
[ 82.319234] DMAR: DRHD: handling fault status reg 3
[ 90.309126] DMAR: DRHD: handling fault status reg 3
[ 90.309132] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc281000 [fault reason 0x05] PTE Write access is not set
[ 90.309137] DMAR: DRHD: handling fault status reg 3
[ 90.309140] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc281000 [fault reason 0x05] PTE Write access is not set
[ 90.309146] DMAR: DRHD: handling fault status reg 3
[ 90.309149] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x dc281000 [fault reason 0x05] PTE Write access is not set
[ 90.309154] DMAR: DRHD: handling fault status reg 3

check find /sys/kernel/iommu_groups/ -type l seems correct
/sys/kernel/iommu_groups/17/devices/0000:08:00.0
/sys/kernel/iommu_groups/17/devices/0000:03:0a.0
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/15/devices/0000:03:03.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:19.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/18/devices/0000:09:00.0
/sys/kernel/iommu_groups/18/devices/0000:03:0b.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:03:08.0
/sys/kernel/iommu_groups/6/devices/0000:00:1a.0
/sys/kernel/iommu_groups/14/devices/0000:03:02.0
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.3
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4

check lspci -nnk seems correct
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
Subsystem: Hewlett-Packard Company Xeon E3-1200 v3 Processor DRAM Controller [103c:1998]
Kernel modules: ie31200_edac
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:040a] (rev 06)
DeviceName: Onboard IGD
Subsystem: Hewlett-Packard Company Xeon E3-1200 v3 Processor Integrated Graphics Controller [103c:1998]
Kernel driver in use: vfio-pci
Kernel modules: i915
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: mei_me
Kernel modules: mei_me
00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller [8086:8c3d] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: serial
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 04)
DeviceName: Onboard LAN
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: e1000e
Kernel modules: e1000e
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d4)
Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d4)
Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1f.0 ISA bridge [0601]: Intel Corporation Q87 Express LPC Controller [8086:8c4e] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: vfio-pci
Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: vfio-pci
Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 04)
Subsystem: Hewlett-Packard Company EliteDesk 800 G1 [103c:1998]
Kernel driver in use: vfio-pci
Kernel modules: i2c_i801
02:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:02.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:08.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:0a.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
03:0b.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1812] (rev 01)
Kernel driver in use: pcieport
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:0123]
Kernel driver in use: vfio-pci
Kernel modules: r8169
06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:0123]
Kernel driver in use: vfio-pci
Kernel modules: r8169
08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:0123]
Kernel driver in use: vfio-pci
Kernel modules: r8169
09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:0123]
Kernel driver in use: vfio-pci
Kernel modules: r8169
 
I tried to set the GPU passthru following documentation, adding GRUB- intel_IOMMU=on, video....off， blacklist with i915...snd...，when I reboot the PVE, I got ROUTER system working(IKUAI, OPENWRT), but synology system down(I think it means network adapter can not get IP). so I restore all seeting, ,the system comeback again. I still want to fix the GPU problem to transcode on synology
 
SYS info
root@pve:~# uname -a
Linux pve 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) x86_64 GNU/Linux
 
Maybe your integrated graphics supports GVT-g? Otherwise make sure its not used/touched before staring the VM by early binding it to vfio-pci, and you might need this work-around.
Passthrough of the boot GPU and/or only GPU and/or integrated graphhics in a system is always more difficult and sometimes impossible. Be aware that you won't have a console anymore to login when anything goes wrong.
 
leesteken said:
Maybe your integrated graphics supports GVT-g? Otherwise make sure its not used/touched before staring the VM by early binding it to vfio-pci, and you might need this work-around.
Passthrough of the boot GPU and/or only GPU and/or integrated graphhics in a system is always more difficult and sometimes impossible. Be aware that you won't have a console anymore to login when anything goes wrong.
Click to expand...
thanks for help， I know that the haswell intel CPU is not easy to pass thru, but why I didn't set any IOMMU configure, but the net and SATA still work correctly on the VM, it's different from the documentation, any special setting on 7.2-3?
 
jeff feng said:
thanks for help， I know that the haswell intel CPU is not easy to pass thru, but why I didn't set any IOMMU configure, but the net and SATA still work correctly on the VM, it's different from the documentation, any special setting on 7.2-3?
Click to expand...
I'm not sure what you mean but 7.2 came with intel_iommu enabled by default but this was later reverted. You will need intel_iommu=on after updating your Proxmox.
 
leesteken said:
I'm not sure what you mean but 7.2 came with intel_iommu enabled by default but this was later reverted. You will need intel_iommu=on after updating your Proxmox.
Click to expand...
it mean my GRUB_CMDLINE_LINUX_DEFAULT="quiet" no IOMMU=ON, modprobe.d/x.conf no blacklist,no unsafe interrupts=1, but when I go to web console directly, add pci, including GPU, SATA console, nerwork adapt were all there see pic 1,and all working but GPU see pic 2 .
OK,maybe I still have to add the commands in GRUB again to updating,
 

Attachments

  • 1.jpg
    1.jpg
    142.8 KB · Views: 27
  • 2.jpg
    2.jpg
    59.5 KB · Views: 27
you could try setting the iommu mode to passthrough with ' iommu=pt' on the kernel commandline (in addition to intel_iommu=on)
 
I trid, finally , turned fastboot in BIOS(I'm not sure if this help)
and grub GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on initcall_blacklist=sysfb_init"
everythin seems good
root@pve:~# dmesg | grep -e DMAR -e IOMMU [ 0.007168] ACPI: DMAR 0x00000000D9915598 0000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.007195] ACPI: Reserving DMAR table memory at [mem 0xd9915598-0xd991564f] [ 0.038854] DMAR: IOMMU enabled [ 0.118128] DMAR: Host address width 39 [ 0.118129] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.118134] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.118136] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.118139] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.118141] DMAR: RMRR base: 0x000000d988c000 end: 0x000000d9898fff [ 0.118143] DMAR: RMRR base: 0x000000dc000000 end: 0x000000de1fffff [ 0.118145] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.118147] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 [ 0.118148] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.118723] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 0.402297] DMAR: No ATSR found [ 0.402298] DMAR: No SATC found [ 0.402300] DMAR: IOMMU feature pgsel_inv inconsistent [ 0.402301] DMAR: IOMMU feature sc_support inconsistent [ 0.402302] DMAR: IOMMU feature pass_through inconsistent [ 0.402303] DMAR: dmar0: Using Queued invalidation [ 0.402309] DMAR: dmar1: Using Queued invalidation [ 0.499498] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 83.319321] DMAR: DRHD: handling fault status reg 3 [ 83.319344] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc200000 [fault reason 0x05] PTE Write access is not set [ 83.319360] DMAR: DRHD: handling fault status reg 3 [ 83.319362] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc200000 [fault reason 0x05] PTE Write access is not set [ 83.319365] DMAR: DRHD: handling fault status reg 3 [ 83.319367] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc200000 [fault reason 0x05] PTE Write access is not set [ 83.319373] DMAR: DRHD: handling fault status reg 3 [ 88.319675] DMAR: DRHD: handling fault status reg 3 [ 88.319681] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc55a000 [fault reason 0x05] PTE Write access is not set [ 88.319687] DMAR: DRHD: handling fault status reg 3 [ 88.319689] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc55a000 [fault reason 0x05] PTE Write access is not set [ 88.319714] DMAR: DRHD: handling fault status reg 3 [ 88.319716] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0xdc55a000 [fault reason 0x05] PTE Write access is not set [ 88.319719] DMAR: DRHD: handling fault status reg 3
 
dcsapak said:
you could try setting the iommu mode to passthrough with ' iommu=pt' on the kernel commandline (in addition to intel_iommu=on)
Click to expand...
I trid , with that, my VM of synology could not get the nerwork adapt work , so i remove that, and now the IOMMU working ,with a little error above, I'm working on that error now. anyway all VMs running well, only the GPU decoding on E3 1265l v3 still fail, working on it too.
hope some one could give some experience on the same hardware
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back