PCI passthrough (attempt 2) with Nvidia GTX 650ti Boost

[pookieman]

Back again .. this forum has been super helpful. Hoping for one more win. I was attempting PCI pass through for graphics with a different card. But that doesn't support parsec. So got a new card - Nvidia GTX 650 ti boost.. followed a bunch of steps.. this is what I have so far

root@homeserver:~# dmesg | grep -e DMAR -e IOMMU

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.013075] ACPI: DMAR 0x0000000079891C20 000148 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[    0.013130] ACPI: Reserving DMAR table memory at [mem 0x79891c20-0x79891d67]
[    0.283912] DMAR: IOMMU enabled
[    0.657525] DMAR: Host address width 46
[    0.657527] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.657535] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.657539] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    0.657545] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.657549] DMAR: RMRR base: 0x0000007ba54000 end: 0x0000007ba63fff
[    0.657552] DMAR: ATSR flags: 0x0
[    0.657554] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
[    0.657557] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
[    0.657560] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
[    0.657563] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
[    0.657566] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
[    0.657569] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    0.657572] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.657573] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.658237] DMAR-IR: Enabled IRQ remapping in xapic mode
[    2.155815] DMAR: dmar0: Using Queued invalidation
[    2.155835] DMAR: dmar1: Using Queued invalidation
[    2.170758] DMAR: Intel(R) Virtualization Technology for Directed I/O

and

root@homeserver:~# cat /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt video=efifb:off video=vesafb:off vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream

and

root@homeserver:~# cat /etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

and

root@homeserver:~# cat /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

and

root@homeserver:~# lspci -v | grep -i nvidia

81:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti Boost] (rev a1) (prog-if 00 [VGA controller])
        Kernel modules: nvidiafb, nouveau
81:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)

and

root@homeserver:~# lspci -n -s 81:00

81:00.0 0300: 10de:11c2 (rev a1)
81:00.1 0403: 10de:0e0b (rev a1)

and

root@homeserver:~# cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:11c2,10de:0e0b disable_vga=1

and

root@homeserver:~# cat /etc/pve/qemu-server/105.conf

bios: ovmf
boot: order=ide0;ide2;net0
cores: 4
efidisk0: local-zfs:vm-105-disk-1,size=1M
hostpci0: 0000:81:00,pcie=1
ide0: local-zfs:vm-105-disk-0,size=50G
ide2: local:iso/Windows10.iso,media=cdrom
machine: pc-q35-6.0
memory: 4096
name: Gamer1
net0: e1000=4A:E5:BD:6F:94:29,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=19db7dc5-8c12-47f3-a2a2-6bcd713e77ed
sockets: 2
vga: virtio,memory=512
vmgenid: c6829e45-d06a-44f6-8317-02c37e9a4689

this is what i see in the VM

 

[leesteken]

Have you tried setting Display to none and/or enable the Primary GPU option? That last one can be very important for NVidia GPUs.

 

[pookieman]

If I set display to none I just see this in the console



‍

 

[leesteken]

If I set display to none I just see this in the console

View attachment 29951

‍

Ah right, yes ofcourse, but do you have output on a monitor connected to the GPU?

 

[pookieman]

Ah right, yes ofcourse, but do you have output on a monitor connected to the GPU?

No.. the card isn't plugged into a monitor as such, does it need to be?

 

[leesteken]

I think that passthrough of NVidia GPUs works better when you enable the Primary GPU option, which I think also disables the emulated display and does require a physical monitor. But maybe it is not required, I used to have problems with a NVidia and switched to AMD for passthrough, so I have little experience.

 

[pookieman]

Tried that too - still the same