[SOLVED] Some games are not working inside a VM with Nvidia GPU

[DevilhandX]

Long story short:
I set up a VM and passed trough a 1080Ti. (It's connected to a monitor)
Installed 8 random Steam games.
Only 2 worked.

Batman: Arkham Asylum Game of the Year Edition - I click on the play button (in launcher), nothing happens, possible crash on startup, no Event Viewer entry.
Friday the 13th: The Game - I click on the play button, shows splash screen, then quits with Bad Image errors (x3daudio1_7.dll, xapofx1_5.dll), no Event Viewer entry.
Hydrophobia: Prophecy - I click on the play button, nothing happens, possible crash on startup, no Event Viewer entry.
Just Cause 2 - I click on the play button, screen turns black then quits, no Event Viewer entry.
Lost Horizon - Works!
Mad Max - Works!
Split/Second - I click on the play button, screen turns black then quits, Event Viewer shows Application Error, exception code "0xc0000005".
The Vanishing of Ethan Carter - I click on the play button, nothing happens, possible crash on startup, no Event Viewer entry.
(Heaven Benchmark 4.0 - Works!)

Hardware:
MSI Z97 MPower MB
32GB Corsair Vengeance Red Pro
i7-4790K
1080Ti
GT710

Proxmox 6.2-4 settings:
/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

# Split/Second and Mad Max throw BSOD (kernel security check failure) without these:
/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm report_ignored_msrs=0

VM conf:

agent: 1
bios: ovmf
boot: cdn
bootdisk: scsi0
cores: 8
cpu: host
efidisk0: local-lv-thin-1:vm-131-disk-1,size=4M
hostpci0: 01:00,pcie=1,x-vga=1
machine: q35
memory: 8192
name: game-server-test
net0: virtio=C6:A9:6F:FF:D4:6A,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lv-thin-1:vm-131-disk-0,backup=0,cache=writethrough,discard=on,size=256G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=83bdec48-c475-4fcd-90d8-2ed6b6bc7c5e
sockets: 1
vmgenid: 8037af99-9631-4d2e-90cd-2e1298397f69

VM:
Windows 10 iso - 21H1
Latest Nvidia drivers (471.41)

What I have already tried:
Installed the same config (same Windows iso, same Nvidia drivers) on the same physical machine. Every game worked, as it should be.
Installed the same config (same Windows iso, same Nvidia drivers) on a VM with ESXi 7.0. Every game worked, so it should be a Proxmox config issue.
Tried blacklisting drivers, adding vfio modules, hiding cpu, etc., (even though I had no problem passig trough the VGA card.)
Disabled/Enabled the iGPU and Multi-monitor mode in the BIOS.
Tried the whole setup with the GT710, same result, 8/2 games are worked.
Tried older Nvidia drivers (456.71).

Any help would be appreciated.

 

[Dunuin]

Maybe there is a newer BIOS or GPU firmware you can update to?

Edit:
And you could try to change the MSI/MSI-X priority. I needed to increase the MSI of the GPUs Soundcard because my GT710 wasn't outputting any sound. Maybe that would help with your GPU stabilities too.
There was a tool from someone of another forum that could be run in Win10 to change the MSI. Don't know the name or link anymore.

Edit: I think it was the MSI utility v3 from here.

 

[DevilhandX]

Maybe there is a newer BIOS or GPU firmware you can update to?

Edit:
And you could try to change the MSI/MSI-X priority. I needed to increase the MSI of the GPUs Soundcard because my GT710 wasn't outputting any sound. Maybe that would help with your GPU stabilities too.
There was a tool from someone of another forum that could be run in Win10 to change the MSI. Don't know the name or link anymore.

Edit: I think it was the MSI utility v3 from here.

My MB BIOS is the "latest" 1.B (2016).
My GPU (ROG-STRIX-GTX1080TI-O11G-GAMING) has the original factory VBIOS from ASUS (86.02.39.00.23). There is no newer VBIOS on ASUS' website only a "Special Silent Fan VBIOS" from 2017. There is no newer VBIOS in TechPowerUp's VGA BIOS collection either.

I have checked the status of MSI and it was enabled (by Proxmox) until I started the VM, then "lspci -vv" showed the GPU and it's audio device as MSI disabled.
I have added the corresponding Registry keys and using the utility raised the priority to High for both devices.

Verifying in HV:
lspci -vv

01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: vfio-pci

01:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: vfio-pci

Verifying in VM:
Windows also shows both devices in MSI mode as they IRQs became negative.

Unfortunately, no luck, the problem still persist.

 

[DevilhandX]

UPDATE:
In my final despair, I tried to reinstall the OS on the same virtual disk (scsi0, see above) and got an 0x8007045d error (maybe an I/O device error?) when trying to select the VirtIO storage during the Windows Setup. After that I detached and deleted the virtual disk on the Proxmox webui and creted a new one (same specs), installed Windows, latest Nvidia drivers, Steam, and the games, and everything worked as they should.
So maybe it was a corrupted Windows install caused by I/O errors on a faulty virtual disk all along. I dunno.
Lesson learned, if something goes wrong, don't try to fix it or tinker with it for 2 days, just purge the damn thing and start everything from scratch...
Thanks again for everyone trying to help!