First off let me say thank you for helping.
I am bashing my head with this Windows VM annoyance
I figure my config is solid! I have a POPOS-VM with the same gpu pass-through and it works flawlessly. I am able to game, run sunshine, and overclock the gpu. It works, simple.
Windows and all it's bull**** doesn't, all forums that I've tried to find, all say its the config with Grub, virtio, blah blah. if that was true then why does my linux vm work so well.
enough ranting here is my config for everything I can think for the moment.
HARDWARE
CPU - Intel(R) Xeon(R) CPU E5-2697 v4
Motherboard - Gigabyte X99 Designare v1
RAM - 160GB
Boot GPU - AMD R9 270 <- First Slot PCIE x16
Generic USB hub <- Second pcie x1 slot
VM GPU - EVGA RTX2070 Black <- Third Slot PCIE x16
Latest BIOS
/etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intremap=no_x2apic_optout"
GRUB_CMDLINE_LINUX=""
dmesg | grep -e IOMMU
[ 0.340631] DMAR: IOMMU enabled
[ 0.875483] DMAR-IR: IOAPIC id 1 under DRHD base 0xdfffc000 IOMMU 1
[ 7.404509] DMAR: IOMMU feature sc_support inconsistent
[ 7.404510] DMAR: IOMMU feature dev_iotlb_support inconsistent
/etc/modules
vfio
vfio_iommu_type1
vfio_pci
dmesg | grep -i vfio
[ 10.118637] VFIO - User Level meta-driver version: 0.3
[ 10.130330] vfio_pci: add [10de:128b[ffffffff:ffffffff]] class 0x000000/00000000
[ 10.130344] vfio_pci: add [10de:0e0f[ffffffff:ffffffff]] class 0x000000/00000000
[ 79.865528] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none
wns=none
[ 79.865935] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=io+memwns=none
[ 79.866299] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 82.547098] vfio-pci 0000:4e:00.0: enabling device (0100 -> 0103)
dmesg | grep 'remapping'
[ 0.875537] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.876119] DMAR-IR: Enabled IRQ remapping in x2apic mode
/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0
/etc/modprobe.d/vfio.conf
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pci
options vfio-pci ids=10de:1f02,10de:10f9,10de:1ada,10de:1adb disable_vga=1 disable_idle_d3=1
/etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm
dmesg | grep -E "DMAR|IOMMU"
[ 0.009621] ACPI: DMAR 0x00000000858A2188 0000E4 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.009645] ACPI: Reserving DMAR table memory at [mem 0x858a2188-0x858a226b]
[ 0.340694] DMAR: IOMMU enabled
[ 0.875509] DMAR: Host address width 46
[ 0.875511] DMAR: DRHD base: 0x000000dfffd000 flags: 0x0
[ 0.875520] DMAR: dmar0: reg_base_addr dfffd000 ver 1:0 cap 8d2008c10ef0466 ecap f0205b
[ 0.875523] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
[ 0.875527] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.875529] DMAR: RMRR base: 0x00000086e06000 end: 0x00000086e15fff
[ 0.875531] DMAR: ATSR flags: 0x0
[ 0.875532] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
[ 0.875535] DMAR-IR: IOAPIC id 1 under DRHD base 0xdfffc000 IOMMU 1
[ 0.875536] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
[ 0.875537] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.876119] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 7.410683] DMAR: [Firmware Bug]: RMRR entry for device 4e:00.2 is broken - applying workaround <-USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
[ 7.410688] DMAR: [Firmware Bug]: RMRR entry for device 52:00.0 is broken - applying workaround<- USB PCIE HUB for plug n play
[ 7.410690] DMAR: No SATC found
[ 7.410692] DMAR: IOMMU feature sc_support inconsistent
[ 7.410693] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 7.410694] DMAR: dmar0: Using Queued invalidation
[ 7.410698] DMAR: dmar1: Using Queued invalidation
[ 7.415683] DMAR: Intel(R) Virtualization Technology for Directed I/O
WINDOWS CONFIG
/etc/pve/qemu-server/102.conf
agent: 1
args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off,hv_vendor_id=intel
balloon: 0
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 10
cpu: host,flags=+pcid;+hv-evmcs;+aes
efidisk0: Zuper:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:4e:00,device-id=0x1f02,pcie=1,vendor-id=0x10de,x-vga=1
hostpci1: 0000:52:00,pcie=1
ide0: local:iso/virtio-win-0.1.266.iso,media=cdrom,size=707456K
ide2: none,media=cdrom
machine: pc-q35-9.2
memory: 32856
meta: creation-qemu=9.2.0,ctime=1742695851
name: WindowsGPUTEST
net0: e1000=BA:77:23:8D:80:4B,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: Zuper:vm-102-disk-1,discard=on,iothread=1,size=120G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=fded493c-301e-4d5f-8a20-53b8349605b0
sockets: 1
vga: none
vmgenid: 420479e4-1c3e-4d60-9e13-e588a1258b84
Weird Issue tho don't know if its my gpu or what, but i have tried ROMFILE with 2 different ways of dumping the VBIOS from another computer with GPU-Z and on proxmox with nvflash, seems to be the same .rom file. Same out come tho. If i enable the romfile in the conf. I no longer get a boot splash screen. Nothing pops up on the monitor. still able to access console with VNC.
I think its because NVIDIA is detecting a VM and stopping the Driver. after the boot screen the monitor turns off. I have no idea how to spoof a VM. I always get ERROR CODE 43
I really could use everyone's help.
I have been trying for a full week now. I have gotten nowhere.
I am bashing my head with this Windows VM annoyance
I figure my config is solid! I have a POPOS-VM with the same gpu pass-through and it works flawlessly. I am able to game, run sunshine, and overclock the gpu. It works, simple.
Windows and all it's bull**** doesn't, all forums that I've tried to find, all say its the config with Grub, virtio, blah blah. if that was true then why does my linux vm work so well.
enough ranting here is my config for everything I can think for the moment.
HARDWARE
CPU - Intel(R) Xeon(R) CPU E5-2697 v4
Motherboard - Gigabyte X99 Designare v1
RAM - 160GB
Boot GPU - AMD R9 270 <- First Slot PCIE x16
Generic USB hub <- Second pcie x1 slot
VM GPU - EVGA RTX2070 Black <- Third Slot PCIE x16
Latest BIOS
/etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intremap=no_x2apic_optout"
GRUB_CMDLINE_LINUX=""
dmesg | grep -e IOMMU
[ 0.340631] DMAR: IOMMU enabled
[ 0.875483] DMAR-IR: IOAPIC id 1 under DRHD base 0xdfffc000 IOMMU 1
[ 7.404509] DMAR: IOMMU feature sc_support inconsistent
[ 7.404510] DMAR: IOMMU feature dev_iotlb_support inconsistent
/etc/modules
vfio
vfio_iommu_type1
vfio_pci
dmesg | grep -i vfio
[ 10.118637] VFIO - User Level meta-driver version: 0.3
[ 10.130330] vfio_pci: add [10de:128b[ffffffff:ffffffff]] class 0x000000/00000000
[ 10.130344] vfio_pci: add [10de:0e0f[ffffffff:ffffffff]] class 0x000000/00000000
[ 79.865528] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none
wns=none[ 79.865935] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=io+mem
wns=none[ 79.866299] vfio-pci 0000:4e:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none
wns=none[ 82.547098] vfio-pci 0000:4e:00.0: enabling device (0100 -> 0103)
dmesg | grep 'remapping'
[ 0.875537] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.876119] DMAR-IR: Enabled IRQ remapping in x2apic mode
/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0
/etc/modprobe.d/vfio.conf
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pci
options vfio-pci ids=10de:1f02,10de:10f9,10de:1ada,10de:1adb disable_vga=1 disable_idle_d3=1
/etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm
dmesg | grep -E "DMAR|IOMMU"
[ 0.009621] ACPI: DMAR 0x00000000858A2188 0000E4 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.009645] ACPI: Reserving DMAR table memory at [mem 0x858a2188-0x858a226b]
[ 0.340694] DMAR: IOMMU enabled
[ 0.875509] DMAR: Host address width 46
[ 0.875511] DMAR: DRHD base: 0x000000dfffd000 flags: 0x0
[ 0.875520] DMAR: dmar0: reg_base_addr dfffd000 ver 1:0 cap 8d2008c10ef0466 ecap f0205b
[ 0.875523] DMAR: DRHD base: 0x000000dfffc000 flags: 0x1
[ 0.875527] DMAR: dmar1: reg_base_addr dfffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.875529] DMAR: RMRR base: 0x00000086e06000 end: 0x00000086e15fff
[ 0.875531] DMAR: ATSR flags: 0x0
[ 0.875532] DMAR: RHSA base: 0x000000dfffc000 proximity domain: 0x0
[ 0.875535] DMAR-IR: IOAPIC id 1 under DRHD base 0xdfffc000 IOMMU 1
[ 0.875536] DMAR-IR: HPET id 0 under DRHD base 0xdfffc000
[ 0.875537] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.876119] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 7.410683] DMAR: [Firmware Bug]: RMRR entry for device 4e:00.2 is broken - applying workaround <-USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
[ 7.410688] DMAR: [Firmware Bug]: RMRR entry for device 52:00.0 is broken - applying workaround<- USB PCIE HUB for plug n play
[ 7.410690] DMAR: No SATC found
[ 7.410692] DMAR: IOMMU feature sc_support inconsistent
[ 7.410693] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 7.410694] DMAR: dmar0: Using Queued invalidation
[ 7.410698] DMAR: dmar1: Using Queued invalidation
[ 7.415683] DMAR: Intel(R) Virtualization Technology for Directed I/O
WINDOWS CONFIG
/etc/pve/qemu-server/102.conf
agent: 1
args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off,hv_vendor_id=intel
balloon: 0
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 10
cpu: host,flags=+pcid;+hv-evmcs;+aes
efidisk0: Zuper:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:4e:00,device-id=0x1f02,pcie=1,vendor-id=0x10de,x-vga=1
hostpci1: 0000:52:00,pcie=1
ide0: local:iso/virtio-win-0.1.266.iso,media=cdrom,size=707456K
ide2: none,media=cdrom
machine: pc-q35-9.2
memory: 32856
meta: creation-qemu=9.2.0,ctime=1742695851
name: WindowsGPUTEST
net0: e1000=BA:77:23:8D:80:4B,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: Zuper:vm-102-disk-1,discard=on,iothread=1,size=120G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=fded493c-301e-4d5f-8a20-53b8349605b0
sockets: 1
vga: none
vmgenid: 420479e4-1c3e-4d60-9e13-e588a1258b84
Weird Issue tho don't know if its my gpu or what, but i have tried ROMFILE with 2 different ways of dumping the VBIOS from another computer with GPU-Z and on proxmox with nvflash, seems to be the same .rom file. Same out come tho. If i enable the romfile in the conf. I no longer get a boot splash screen. Nothing pops up on the monitor. still able to access console with VNC.
I think its because NVIDIA is detecting a VM and stopping the Driver. after the boot screen the monitor turns off. I have no idea how to spoof a VM. I always get ERROR CODE 43
I really could use everyone's help.
I have been trying for a full week now. I have gotten nowhere.
Last edited: