GPU passthrough help - nvidia 2070 on Dell XPS

[CAAD]

Hello all,

I am looking for any tips or help in trying other options in the configuration adventure - much appreciated in advance.

I've been going through various tutorials on configuring GPU passthrough with an nVidia GPU 2070 Super.
With an external monitor connected, I tried a mix of VM configs for Windows 10, however with my last attempt, the VM dies on startup.
I've been mainly following the Beginner's Guide on the config so my config closely resembles it with a few tweaks that I thought was useful from other guides.

Any suggestions on what I should try?

The system information is as follows
Dell XPS 8400
8 x Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz (1 Socket)
Kernel Version
Linux 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100)
PVE Manager Version
pve-manager/7.3-3/c3928077

grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on initcall_blacklist=sysfb_init iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff pcie_aspm=off"

modules:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

added per beginner's guide:
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

blacklisted kernel:
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

pci hardware options
options vfio-pci ids=10de:1e84,10de:10f8,10de:lad8,10de:lad9 disable_vga=1

updated grub and executed update-initramfs -u.

VM config:
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cores: 4
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hookscript: local:snippets/gpu-hookscript.sh
hostpci0: 0000:01:00,pcie=1,x-vga=1
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
ide2: local:iso/en_windows_10_multiple_editions_version_1607_updated_jul_2016_x64_dvd_9058187.iso,media=cdrom,size=4277722K
machine: pc-q35-7.1
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
memory: 8192
meta: creation-qemu=7.1.0,ctime=1676723715
name: win10
net0: virtio=EA:57:25:A3:BB:A6,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: local-lvm:vm-101-disk-1,iothread=1,size=120G
scsihw: virtio-scsi-single
smbios1: uuid=47a5da1d-f17c-43bd-a4e1-7467834b9c2c
sockets: 1
tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0
vga: none
vmgenid: f691fd83-2bd0-407d-9262-ea09e3dc5183
hostpci0: 01:00,pcie=1,romfile=2070.rom

syslog when starting the VM:
Feb 18 12:48:39 pve pvedaemon[1937]: start VM 101: UPIDve:00000791:000046D9:63F139A7:qmstart:101:root@pam:
Feb 18 12:48:39 pve kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+memwns=io+mem
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: Removing from iommu group 11
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: [10de:1e84] type 00 class 0x030000
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: reg 0x10: [mem 0xf6000000-0xf6ffffff]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: reg 0x14: [mem 0xe0000000-0xefffffff 64bit pref]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: reg 0x1c: [mem 0xf0000000-0xf1ffffff 64bit pref]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: reg 0x30: [mem 0xf7000000-0xf707ffff pref]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:01.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: vgaarb: setting as boot VGA device
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: Adding to iommu group 11
Feb 18 12:48:39 pve kernel: pcieport 0000:00:1c.0: Enabling MPC IRBNCE
Feb 18 12:48:39 pve kernel: pcieport 0000:00:1c.0: Intel PCH root port ACS workaround enabled
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: BAR 1: assigned [mem 0xe0000000-0xefffffff 64bit pref]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: BAR 3: assigned [mem 0xf0000000-0xf1ffffff 64bit pref]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf6ffffff]
Feb 18 12:48:39 pve kernel: pci 0000:01:00.0: BAR 5: assigned [io 0xe000-0xe07f]
Feb 18 12:48:39 pve kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=nonewns=io+mem
Feb 18 12:48:40 pve systemd[1]: Started 101.scope.
Feb 18 12:48:40 pve systemd-udevd[1940]: Using default interface naming scheme 'v247'.
Feb 18 12:48:40 pve systemd-udevd[1940]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 18 12:48:40 pve kernel: device tap101i0 entered promiscuous mode
Feb 18 12:48:40 pve systemd-udevd[1940]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 18 12:48:40 pve systemd-udevd[1940]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 18 12:48:40 pve systemd-udevd[1939]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 18 12:48:40 pve systemd-udevd[1939]: Using default interface naming scheme 'v247'.
Feb 18 12:48:40 pve kernel: vmbr0: port 2(fwpr101p0) entered blocking state
Feb 18 12:48:40 pve kernel: vmbr0: port 2(fwpr101p0) entered disabled state
Feb 18 12:48:40 pve kernel: device fwpr101p0 entered promiscuous mode
Feb 18 12:48:40 pve kernel: vmbr0: port 2(fwpr101p0) entered blocking state
Feb 18 12:48:40 pve kernel: vmbr0: port 2(fwpr101p0) entered forwarding state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Feb 18 12:48:40 pve kernel: device fwln101i0 entered promiscuous mode
Feb 18 12:48:40 pve kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 1(fwln101i0) entered forwarding state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 2(tap101i0) entered blocking state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 2(tap101i0) entered disabled state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 2(tap101i0) entered blocking state
Feb 18 12:48:40 pve kernel: fwbr101i0: port 2(tap101i0) entered forwarding state
Feb 18 12:48:41 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Feb 18 12:48:41 pve kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Feb 18 12:48:41 pve kernel: resource sanity check: requesting [mem 0x000c0000-0x000dffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000dffff window]
Feb 18 12:48:41 pve kernel: caller pci_map_rom+0x7f/0x1f0 mapping multiple BARs
Feb 18 12:48:41 pve kernel: vfio-pci 0000:01:00.0: No more image in the PCI ROM
Feb 18 12:48:41 pve kernel: fwbr101i0: port 2(tap101i0) entered disabled state
Feb 18 12:48:41 pve kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Feb 18 12:48:41 pve kernel: vmbr0: port 2(fwpr101p0) entered disabled state
Feb 18 12:48:41 pve kernel: device fwln101i0 left promiscuous mode
Feb 18 12:48:41 pve kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Feb 18 12:48:41 pve kernel: device fwpr101p0 left promiscuous mode
Feb 18 12:48:41 pve kernel: vmbr0: port 2(fwpr101p0) entered disabled state
Feb 18 12:48:41 pve pvedaemon[1110]: VM 101 qmp command failed - VM 101 not running
Feb 18 12:48:41 pve pvedaemon[1108]: VM 101 qmp command failed - VM 101 not running
Feb 18 12:48:41 pve pvedaemon[1947]: stopping swtpm instance (pid 1955) due to QEMU startup error
Feb 18 12:48:41 pve pvedaemon[1937]: start failed: QEMU exited with code 1
Feb 18 12:48:41 pve pvedaemon[1108]: <root@pam> end task UPIDve:00000791:000046D9:63F139A7:qmstart:101:root@pam: start failed: QEMU exited with code 1
Feb 18 12:48:41 pve pvedaemon[1110]: <root@pam> starting task UPIDve:000007F5:000047BF:63F139A9:vncproxy:101:root@pam:
Feb 18 12:48:41 pve pvedaemon[2037]: starting vnc proxy UPIDve:000007F5:000047BF:63F139A9:vncproxy:101:root@pam:
Feb 18 12:48:41 pve pveproxy[1117]: proxy detected vanished client connection
Feb 18 12:48:42 pve systemd[1]: 101.scope: Succeeded.
Feb 18 12:48:42 pve systemd[1]: 101.scope: Consumed 1.552s CPU time.
Feb 18 12:48:42 pve pvedaemon[1912]: connection timed out
Feb 18 12:48:42 pve pvedaemon[1110]: <root@pam> end task UPIDve:00000778:00004420:63F139A0:vncproxy:101:root@pam: connection timed out
Feb 18 12:48:51 pve pvedaemon[2037]: connection timed out
Feb 18 12:48:51 pve pvedaemon[1110]: <root@pam> end task UPIDve:000007F5:000047BF:63F139A9:vncproxy:101:root@pam: connection timed out

 

[CAAD]

I have solved the problem of the 2070 Super not showing or crashing the system.
The Windows 10 version I was using is too old . Need at least Windows 10 "2004" build 2H22.
Since I am using a Dell, they have a link to create a Windows 10 ISO in a solution answer.

 

[CAAD]

I got the code 43 error but was able to install the latest Nvidia drivers for the 2070 Super.
I had to delete the device entry in "Device Manager" and let it rescan. It then added it as "VGA compatible adapter".
When I looked at the properties of the "VGA adapter" it said Nvidia and the system recognized it without the 43 error.

I had to dump and add the rom to this line to get rid of the code 43 from repeating.
hostpci0: 0000:01:00,pcie=1,rombar=0,x-vga=1,romfile=2070.rom

 

[CAAD]

Further research of the code 43 error in the WIndows VM led me to this great thread.
https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/post-471013

The step by step instructions of removing BOOTFB in the address space with the error below was the final fix!

pve kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]

Thanks to
@StephenM64 for the script
@nick.kopas for making the hookscript easy
@Lefuneste for showing what to look for in iomem

My monitor lit up and is now connected to the Windows VM.
Need to kick the device to get the old code 43 error out of the way by uninstalling device and rediscovering.

 

[leesteken]

Was using initcall_blacklist=sysfb_init instead of (the no longer working) nofb nomodeset video=vesafb:off video=efifb:off , as shown later in that thread, not working for you?

 

[CAAD]

Was using initcall_blacklist=sysfb_init instead of (the no longer working) nofb nomodeset video=vesafb:off video=efifb:off , as shown later in that thread, not working for you?

I was happy to have something working after an exhausting search for a fix, so I didn’t read further but thanks for the tip, I will try this!
I think this needs a FAQ, so much old info out there and version specific fixes.

 

[CAAD]

Was using initcall_blacklist=sysfb_init instead of (the no longer working) nofb nomodeset video=vesafb:off video=efifb:off , as shown later in that thread, not working for you?

Indeed it helps! I can't be sure but my GPU seems to have lower load (better performance?) according to the task manager->performance->Nvidia GPU after this?

I took out the following:
1. Hook script from the <VMID>.conf
2. Removed a bunch of "video=" statements in my grub.

My final grub line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:1e84 pcie_acs_override=downstream,multifunction vfio_pci.disable_vga=1 textonly initcall_blacklist=sysfb_init"

Inspected the /proc/iomem for the BAR 3 memoroy reservation and indeed 01.00.0 is clear (where my 2070 GPU is).
Tested Diablo Immortal over RDP app on a laptop running Windows 10 VM. It is working great!

Thanks @leesteken for the tip!

 

[leesteken]

IGRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:1e84 pcie_acs_override=downstream,multifunction vfio_pci.disable_vga=1 textonly initcall_blacklist=sysfb_init"

I think that you also don't need textonly nor pcie_acs_override=downstream,multifunction. And nobody really needs iommu=pt.
You might want to bind the audio function of the GPU to vfio-pci as well and add like this vfio-pci.ids=10de:1e84,xyzw:abcd (lookup the actual ID with lspci -nns 01:00.1).
Anyway, just some tips to dot the i's and cross-out some t's. Glad you got it working by yourself (and sorry for not spotting this on your first post).

 

[CAAD]

I think that you also don't need textonly nor pcie_acs_override=downstream,multifunction. And nobody really needs iommu=pt.
You might want to bind the audio function of the GPU to vfio-pci as well and add like this vfio-pci.ids=10de:1e84,xyzw:abcd (lookup the actual ID with lspci -nns 01:00.1).
Anyway, just some tips to dot the i's and cross-out some t's. Glad you got it working by yourself (and sorry for not spotting this on your first post).

Cleaned up the line and heres my update. Works!

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio-pci.ids=10de:1e84,10de:10f8 vfio_pci.disable_vga=1 initcall_blacklist=sysfb_init"