Windows 11 VM won't boot with RTX 3090 Passthrough

[raddacle]

I just got a RTX 3090 w/24GB GPU to passthrough to a Windows 11 VM, and I installed the drivers on the Proxmox host, though the Windows VM no longer boots after passing through the GPU. The start task just hangs indefinitely, and the console never turns on. I'm fairly new to Proxmox, so I'm still learning all the ropes, but I've been unsuccessful diagnosing the issue. Any advice?

I have VT-d and Resizable BAR enabled in the host BIOS (Gigabyte Z590 Aorus Pro AX).

I've tried enabling/disabling "All Functions", "Primary GPU", and "ROM Bar" to no success.

Proxmox version info:

Proxmox VE Version: 8.1.0
Running Kernel: 6.5.11-7-pve
pve-manager: 8.1.3
proxmox-kernel-6.5: 6.5.11-7
pve-qemu-kvm: 8.1.2-5
qemu-server: 8.0.10
zfsutils-linux: 2.2.2-pve1
NVIDIA Driver Version: 525.147.05
CUDA Version: 12.0

Here's the VM's config:

agent: 1
bios: ovmf
boot: order=scsi0;scsi1
cores: 6
cpu: x86-64-v2-AES
efidisk0: Data:vm-106-disk-0,efitype=4m,size=1M
hostpci0: 0000:01:00.0,rombar=0
machine: pc-q35-8.1
memory: 32768
meta: creation-qemu=8.1.2,ctime=1702236078
name: windows
net0: e1000=BC:24:11:B5:55:15,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: Data:vm-106-disk-1,iothread=1,size=200G
scsi1: local:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
scsihw: virtio-scsi-single
smbios1: uuid=98ab512d-3626-44c6-bdee-eac25c991359
sockets: 1
tpmstate0: Data:vm-106-disk-2,size=4M,version=v2.0
vmgenid: ce1853df-4031-4182-b3f4-61ec5495d860

GRUB file:

GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=vesafb:off video=efi>
GRUB_CMDLINE_LINUX=""

 

[chchia]

i saw you also posted this in reddit.

anyway i believe this is possible, although i dont have 3090, but previously i do have a 3060 passthrough to a windows 11 machine and working perfectly with HDMI and audio output to my AV Receiver to TV.

i recently changed the 3060 to 2060, but there is no change on the proxmox host configuration and VM configuration, let me share my setting:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intel_pstate=disable"

this is in grub, i think you may ignore pstate as i am using non default CPU governor, if you interested you can search my previous post.

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist i2c_nvidia_gpu

this is mobprode.d blacklist

options vfio-pci ids=8086:7ae0,8086:7ae2,10de:2503,10de:228e,10ec:5763,1cc1:8201,1b21:0612,10de:21c4,10de:1aeb,10de:1aec,10de:1aed

these is vfio, but seriously i think VFIO doesnt change much, as actually i did not add to VFIO for my 2060 graphic card yet it is still working correctly. my 2060 are 10de:1f08 and as you can see it is not inside VFIO yet it is working perfectly.


this is the VM setup

balloon: 0
bios: ovmf
boot: order=virtio0
cores: 6
cpu: host
cpuunits: 500
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,x-vga=1,romfile=3060.bin
hostpci1: 0000:06:00.0,pcie=1
ide2: none,media=cdrom
machine: pc-q35-8.1
memory: 16384
meta: creation-qemu=7.1.0,ctime=1673028164
name: HTPC
net0: virtio=4A:1F:28:xxxxx  ,bridge=vmbr0,mtu=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=xxxxxxxx
sockets: 1
startup: order=2,up=10
tablet: 0
tpmstate0: local-lvm:vm-102-disk-1,size=4M,version=v2.0
usb0: host=1-3.2.1
usb1: host=1-3.2.2
vga: none
virtio0: local-lvm:vm-102-disk-2,backup=0,cache=none,discard=on,replicate=0,size=100G
vmgenid: xxxxxx

i have the 2060 at 01:00 passthrough to windows 11, and it is still using my previous romfile from 3060, and yet it is still working correctly.

in summary i cant see any reason why it cant boot.

please install a standard display driver together with your 3090, so at least you get some screen output from proxmox noVNC and understand whats going on.

 

[leesteken]

I have VT-d and Resizable BAR enabled in the host BIOS (Gigabyte Z590 Aorus Pro AX).

Resizable BAR is not (yet) support by KVM/QEMU/Proxmox, so you'll need to disable it in the motherboard BIOS.

I just got a RTX 3090 w/24GB GPU to passthrough to a Windows 11 VM, and I installed the drivers on the Proxmox host, though the Windows VM no longer boots after passing through the GPU. The start task just hangs indefinitely, and the console never turns on. I'm fairly new to Proxmox, so I'm still learning all the ropes, but I've been unsuccessful diagnosing the issue. Any advice?

Set the VM Display to None to force the use of the GPU and make sure to attach a physical display. Make sure the GPU is not used by the Proxmox host during boot of the system. Also try starting the VM with less memory (8GB for example) as all VM memory must be pinned in to host RAM because of the PCI(e) passthrough.
Are there not error messages in journalctl (around the time of starting the VM)? Are there no errors in the Windows Event logs when it starts without passthrough (about the start with passthrough)?

 

[raddacle]

Resizable BAR is not (yet) support by KVM/QEMU/Proxmox, so you'll need to disable it in the motherboard BIOS.

Set the VM Display to None to force the use of the GPU and make sure to attach a physical display. Make sure the GPU is not used by the Proxmox host during boot of the system. Also try starting the VM with less memory (8GB for example) as all VM memory must be pinned in to host RAM because of the PCI(e) passthrough.
Are there not error messages in journalctl (around the time of starting the VM)? Are there no errors in the Windows Event logs when it starts without passthrough (about the start with passthrough)?

I disabled Resizable BAR, uninstalled the drivers from Proxmox host, and set the VM to 8GB of RAM- with all that set it boots to Windows! If I set the RAM back to 32GB, it crashes. How can I properly assign more RAM to the VM then?

 

[leesteken]

I disabled Resizable BAR, uninstalled the drivers from Proxmox host, and set the VM to 8GB of RAM- with all that set it boots to Windows! If I set the RAM back to 32GB, it crashes. How can I properly assign more RAM to the VM then?

Make sure Proxmox has well over 32GB of memory free (at least 36G or so)? Make sure to start the VM after a reboot from the Proxmox host, when memory is not yet fragmented?
By the way, what do you mean by "it crashes".