Windows 11 VM with RTX 5090 PCIe Passthrough Fails to Boot with >512 GB RAM

superviper

New Member
Oct 22, 2023
2
0
1
System Overview:


  • Host CPUs: Dual-socket AMD EPYC 9554 (256 cores total)
  • Total RAM: 1 TB ECC RDIMM
  • GPU: NVIDIA RTX 5090 (passed through via VFIO)
  • Proxmox Version: pve-manager/8.4.0/ec58e45e1bcdf2ac
  • Kernel: 6.8.12-9-pve
  • QEMU Version: (from /usr/bin/qemu-system-x86_64 --version – please confirm exact version if needed)



Problem Description:


I’m experiencing a critical issue when assigning high RAM (>512 GB) to a Windows 11 Pro VM with GPU passthrough enabled. The VM hangs or fails to boot properly when the following conditions are met:


  • RAM assigned > 512 GB
  • PCI passthrough (hostpci0) enabled (for RTX 5090)

When GPU passthrough is disabled, the VM can boot successfully with up to 1 TB RAM. So the issue only occurs when both high memory and PCI passthrough are enabled together.


Attempting to increase the MMIO reservation using this line in the config:


args: -fw_cfg name=opt/ovmf/PcdPciMmio64Size,value=0x200000000

fails with:

kvm: -fw_cfg name=opt/ovmf/PcdPciMmio64Size,value=...: Invalid parameter 'value'


This seems to be due to a limitation in the default OVMF/QEMU build on Proxmox 8.4, where setting this parameter is not supported.




VM Configuration (/etc/pve/qemu-server/108.conf):


agent: enabled=1
balloon: 0
bios: ovmf
boot: order=ide0;ide2;net0
cores: 127
cpu: EPYC-Genoa
efidisk0: local-lvm:vm-108-disk-3,efitype=4m,pre-enrolled-keys=1,size=128M
hostpci0: 0000:c1:00,pcie=1
ide0: local-lvm:vm-108-disk-1,size=62G
ide2: local:iso/virtio-win.iso,media=cdrom,size=709474K
machine: pc-q35-9.2+pve1
memory: 786432
meta: creation-qemu=9.2.0,ctime=1747692088
name: MaciejVM2
net0: e1000=BC:24:11:13:80:31,bridge=vmbr0,firewall=1
numa: 1
numa0: cpus=0-126,mem=393216
numa1: cpus=127-253,mem=393216
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=1af339b5-a15d-42a0-8462-28244359ff1b
sockets: 2
tpmstate0: local-lvm:vm-108-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: 90a29539-fdaa-4f27-979a-6931e932fe8d


(Previously attempted to add args: -fw_cfg name=opt/ovmf/PcdPciMmio64Size,value=0x200000000 – causes QEMU to exit with Invalid parameter 'value')




What I’ve Confirmed:


  • VM boots fine with ≤512 GB RAM and PCI passthrough enabled
  • VM boots fine with 1 TB RAM if PCI passthrough is disabled
  • Enabling both causes boot failure
  • Hugepages enabled and NUMA properly balanced across two nodes
  • GPU works under passthrough in other VMs with less memory



Expected Behavior:


A Windows 11 VM should boot successfully with PCI passthrough (RTX 5090) and high RAM (768 GB – 1 TB), assuming memory and PCI layout are valid. This may require increased MMIO space in the guest firmware.




Questions:


  1. Is there a supported way in Proxmox 8.4 to run Windows system on Proxmox with passthrough RTX5090 and RAM memory >512GB?



Please let me know if additional logs or test configs would be helpful. Thanks for your support.
 
Last edited: