GPU passthrough

Djin · Aug 14, 2018

Hello,
Already, thank you for the great tool that is Proxmox.

I am currently on a home server on vSphere and I want to switch to Proxmox.
I need to be able to use the GPU for some jobs and vSphere does not hide the virtualization from the NVidia driver.
My main goal is not to display the output of the VM on a screen but to use the power of the GPU for work in a VM.
If you can display the VM on a screen I'm interested but it's secondary (I understand that yes).
That's why I want to use Proxmox.

And of course I have problems (this can be explained by my lack of experience).

My material:

Motherboard: MSI z77a-gd65
Ram: 2x8GB
CPU: Intel I7 3770
HDD:
1 128GB usb stick for Proxmox installation, ISO and template
1 240GB SSD for VMs
GPU : GTX770 and Intel HD Graphics 4000

Information that I could see on the internet, my motherboard manages Vt-d.
So I activated it in the BIOS.
I also activated the UEFI boot because I understood that it was mandatory.

Have I just until then?

Your documentation indicates that you must edit grub by replacing the line GRUB_CMDLINE_LINUX_DEFAULT = "quiet"
By
GRUB_CMDLINE_LINUX_DEFAULT = "quiet intel_iommu = on"

After update-grub, unfortunately, the command dmesg | grep -e DMAR -e IOMMU does not give me any results.

Did I miss something?

Thank you in advance for your help.

Djin · Aug 23, 2018

Hello,

I'm coming back with my personal server.

Like many, I followed the recommendations of this post:
https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/

But in Windows 10, I have the famous error 43.

Could you help me?

Here are the details of my configuration:

Code:

root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000BD47A518 0000B8 (v01 INTEL  SNB      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.004000] DMAR: Host address width 36
[    0.004000] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.004000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.004000] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.004000] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.004000] DMAR: RMRR base: 0x000000bd2e5000 end: 0x000000bd2f2fff
[    0.004000] DMAR: RMRR base: 0x000000bf800000 end: 0x000000cf9fffff
[    0.004000] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.004000] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.004000] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.004000] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.522548] DMAR: No ATSR found
[    0.522580] DMAR: dmar0: Using Queued invalidation
[    0.522584] DMAR: dmar1: Using Queued invalidation
[    0.586156] DMAR: Software identity mapping for device 0000:00:00.0
[    0.586158] DMAR: Software identity mapping for device 0000:00:01.0
[    0.586159] DMAR: Software identity mapping for device 0000:00:01.1
[    0.586163] DMAR: Software identity mapping for device 0000:00:02.0
[    0.586164] DMAR: Software identity mapping for device 0000:00:14.0
[    0.586165] DMAR: Software identity mapping for device 0000:00:16.0
[    0.586166] DMAR: Software identity mapping for device 0000:00:19.0
[    0.586167] DMAR: Software identity mapping for device 0000:00:1a.0
[    0.586168] DMAR: Software identity mapping for device 0000:00:1b.0
[    0.586169] DMAR: Software identity mapping for device 0000:00:1d.0
[    0.586170] DMAR: Software identity mapping for device 0000:00:1f.0
[    0.586170] DMAR: Software identity mapping for device 0000:00:1f.2
[    0.586171] DMAR: Software identity mapping for device 0000:00:1f.3
[    0.586175] DMAR: Software identity mapping for device 0000:01:00.0
[    0.586176] DMAR: Software identity mapping for device 0000:01:00.1
[    0.586180] DMAR: Software identity mapping for device 0000:02:00.0
[    0.586181] DMAR: Setting RMRR:
[    0.586182] DMAR: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.587198] DMAR: Setting identity map for device 0000:00:14.0 [0xbd2e5000 - 0xbd2f2fff]
[    0.587200] DMAR: Setting identity map for device 0000:00:1a.0 [0xbd2e5000 - 0xbd2f2fff]
[    0.587201] DMAR: Setting identity map for device 0000:00:1d.0 [0xbd2e5000 - 0xbd2f2fff]
[    0.587202] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.587203] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.587543] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.596823] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[    0.596857] DMAR: Setting identity map for device 0000:00:1a.0 [0xbd2e5000 - 0xbd2f2fff]
[    0.616430] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[    0.616463] DMAR: Setting identity map for device 0000:00:1d.0 [0xbd2e5000 - 0xbd2f2fff]
[    0.714549] DMAR: 32bit 0000:02:00.0 uses non-identity mapping
[    8.583242] [drm] DMAR active, disabling use of stolen memory

Code:

lspci -v

root@pve:~# lspci -v

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
        Flags: bus master, fast devsel, latency 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>
        Kernel driver in use: ivb_uncore
        Kernel modules: ie31200_edac

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: f6000000-f70fffff
        Prefetchable memory behind bridge: 00000000e0000000-00000000e9ffffff
        Capabilities: [88] Subsystem: Micro-Star International Co., Ltd. [MSI] Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [a0] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [140] Root Complex Link
        Capabilities: [d94] #19
        Kernel driver in use: pcieport
        Kernel modules: shpchp

00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        Memory behind bridge: f7800000-f78fffff
        Capabilities: [88] Subsystem: Micro-Star International Co., Ltd. [MSI] Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [a0] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [140] Root Complex Link
        Capabilities: [d94] #19
        Kernel driver in use: pcieport
        Kernel modules: shpchp

00:02.0 Display controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
        Flags: bus master, fast devsel, latency 0, IRQ 31
        Memory at f7400000 (64-bit, non-prefetchable) [size=4M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at f000 [size=64]
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [a4] PCI Advanced Features
        Kernel driver in use: i915
        Kernel modules: i915

00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C210 Series Chipset Family USB xHCI Host Controller
        Flags: bus master, medium devsel, latency 0, IRQ 26
        Memory at f7920000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [70] Power Management version 2
        Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
        Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C216 Chipset Family MEI Controller
        Flags: bus master, fast devsel, latency 0, IRQ 30
        Memory at f793a000 (64-bit, non-prefetchable) [size=16]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Kernel driver in use: mei_me
        Kernel modules: mei_me

00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 04)
        Subsystem: Micro-Star International Co., Ltd. [MSI] 82579V Gigabit Network Connection
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at f7900000 (32-bit, non-prefetchable) [size=128K]
        Memory at f7938000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at f080 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [e0] PCI Advanced Features
        Kernel driver in use: e1000e
        Kernel modules: e1000e

00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C216 Chipset Family USB Enhanced Host Controller
        Flags: bus master, medium devsel, latency 0, IRQ 16
        Memory at f7937000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)

        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C216 Chipset Family High Definition Audio Controller
        Flags: fast devsel, IRQ 4
        Memory at f7930000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [50] Power Management version 2
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [130] Root Complex Link
        Kernel modules: snd_hda_intel

00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C216 Chipset Family USB Enhanced Host Controller
        Flags: bus master, medium devsel, latency 0, IRQ 23
        Memory at f7936000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Z77 Express Chipset LPC Controller
        Flags: bus master, medium devsel, latency 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich

00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 29
        I/O ports at f0d0 [size=8]
        I/O ports at f0c0 [size=4]
        I/O ports at f0b0 [size=8]
        I/O ports at f0a0 [size=4]
        I/O ports at f060 [size=32]
        Memory at f7935000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ahci
        Kernel modules: ahci

00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
        Subsystem: Micro-Star International Co., Ltd. [MSI] 7 Series/C216 Chipset Family SMBus Controller
        Flags: medium devsel, IRQ 18
        Memory at f7934000 (64-bit, non-prefetchable) [size=256]
        I/O ports at f040 [size=32]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] GK104 [GeForce GTX 770]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GK104 HDMI Audio Controller
        Flags: fast devsel, IRQ 10
        Memory at f7080000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

02:00.0 FireWire (IEEE 1394): LSI Corporation FW643 [TrueFire] PCIe 1394b Controller (rev08) (prog-if 10 [OHCI])
        Subsystem: Device 5901:1101
        Flags: bus master, fast devsel, latency 0, IRQ 27
        Memory at f7800000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Power Management version 3
        Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [170] Device Serial Number 01-08-00-00-00-01-8d-04
        Kernel driver in use: firewire_ohci
        Kernel modules: firewire_ohci

Code:

root@pve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist amdgpu
blacklist snd_hda_intel
blacklist pcspkr

Code:

root@pve:~# lspci -n -s 01:00
01:00.0 0300: 10de:1184 (rev a1)
01:00.1 0403: 10de:0e0a (rev a1)

Code:

root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1184,10de:0e0a disable_vga=1

Code:

nano /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: dc
bootdisk: scsi1
cores: 1
hostpci0: 01:00.0,pcie=1,x-vga=on
cpu: host,hidden=1
freeze: 1
ide0: local:iso/virtio-win-0.1.149.iso,media=cdrom,size=316634K
machine: q35
memory: 4096
name: win10
net0: virtio=FE:80:DA:1C:49:60,bridge=vmbr0
numa: 1
ostype: win10
scsi1: SSD:100/vm-100-disk-1.qcow2,cache=writethrough,size=50G
scsihw: virtio-scsi-pci
smbios1: uuid=bf3f421c-670e-4fcf-9435-016476e788fc
sockets: 4

AlexLup · Aug 23, 2018

Hi,
I managed to get my HP Z400 passthrough my Nvidia 750ti (Server 2016/BIOS not UEFI) by following this guide: https://github.com/sk1080/nvidia-kvm-patcher

You will basically set the server into "test mode", then patch an old driver with the above and it works.
Oh and putting in another card and dumping the GFX firmware was a must as well, the firmwares found on techcrunch where no good..

If anyone has an easier way please post here!!

Here is are configs:

Code:

/etc/pve/qemu-server/100.conf

#/usr/share/kvm/gtx750ti.rom
agent: 1
balloon: 0
boot: dc
bootdisk: sata0
cores: 6
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=123456789abc,kvm=off'
hostpci0: 0f:00.0,pcie=1,x-vga=on,romfile=gtx750ti.rom
hostpci1: 0f:00.1,pcie=1
machine: q35
memory: 8096
name: Plex
net0: virtio=D6:78:0E:6D:EC:DD,bridge=vmbr0
net1: virtio=26:7B:B3:D5:1B:0F,bridge=vmbr1
numa: 0
onboot: 1
ostype: win10
parent: Working_nVidia_driver_v3
protection: 1
scsi0: ceph_pool_vm:vm-100-disk-1,cache=directsync,discard=on,iothread=1,size=182G
scsihw: virtio-scsi-pci
smbios1: uuid=960377eb-babf-4bb8-9703-f2805c4ffd8c
sockets: 1
startup: order=3

Code:

/etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

Code:

/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Code:

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

Code:

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1380,10de:0fbc

Code:

/etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Code:

/etc/default/grub (dont forget to update-grub after editing)
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:eek:ff"
...

AlexLup · Nov 4, 2018

An update; Installing a UEFI windows with GTX 1050 Ti + UEFI BIOS actually works very ok without the nVidia patcher.

Well, until it stops all togehter, but I suspect its my new kernel or the OS loads the GFX card to early..

mercury131 · Nov 5, 2018

AlexLup said:
An update; Installing a UEFI windows with GTX 1050 Ti + UEFI BIOS actually works very ok without the nVidia patcher.

Well, until it stops all togehter, but I suspect its my new kernel or the OS loads the GFX card to early..

Hello!

Can you post your VM config and proxmox configs?

AlexLup · Nov 6, 2018

HP Z400 passthrough my Nvidia 1050 ti (Windows 10 Pro /UEFI BIOS)

Code:

/etc/pve/qemu-server/100.conf

# ROM file at /usr/share/kvm/Gtx1050ti.rom, extracted with romparser
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=1234567890a4,kvm=off'
balloon: 3500
bios: ovmf
boot: dc
bootdisk: scsi0
cores: 6
cpu: host
efidisk0: ceph_pool_vm:base-116-disk-1/vm-117-disk-0,size=1M
hostpci0: 28:00.0,pcie=1,romfile=Gtx1050ti.rom
hostpci1: 28:00.1,pcie=1
hostpci2: 00:1b.0 # Got rid of the chopping sound by passing hosts audio
ide0: none,media=cdrom
machine: q35
memory: 6096
name: Gaming-Station
net0: virtio=BA:F0:3D:30:CA:27,bridge=vmbr0
numa: 1
ostype: win10
parent: Working
scsi0: ceph_pool_vm:base-116-disk-0/vm-117-disk-1,cache=writeback,discard=on,iothread=1,size=110G
scsi1: ceph_pool_vm:vm-100-disk-0,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=150G
scsi2: ceph_pool_vm:vm-100-disk-2,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=150G
scsi3: ceph_pool_vm:vm-100-disk-3,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=150G
scsi4: ceph_pool_vm:vm-100-disk-4,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=150G
scsihw: virtio-scsi-single
smbios1: uuid=f55931e1-593e-498a-8343-1aea8b227332
sockets: 1

Code:

/etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

Code:

/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Code:

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

Code:

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c82,10de:0fb9,8086:3a3e disable_vga=1

Code:

/etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Code:

/etc/default/grub (dont forget to update-grub after editing)
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off"
...

Please note that after an update this stopped working without me changing a thing, troubleshooting is still ongoing!

Search

Search

GPU passthrough

Djin

New Member

Djin

New Member

AlexLup

Well-Known Member

AlexLup

Well-Known Member

mercury131

Active Member

AlexLup

Well-Known Member