Unable to boot second Windows VM (GPU passthrough)

[remz1337]

TLDR: I have 1 working Windows 11 VM with GPU passthrough. Tried to create a new VM with same config but the new one doesn't boot.

About 2 years ago I created a Windows 11 VM with GPU passthrough (Followed every possible blog/thread out there to get it working) and I've been using it since (It's off most of the time, I start it once in a while when I want to play some games). Now, I'm working on automating the deployment of said VM, but the new VM does not start. I see this message in the noVNC console: "guest has not initialized the display (yet)".

The only way I was able to make it work was changing the machine type from q35 to i440fx, but then I lose my GPU passthrough.

Using PVE 7.4-18

Any idea why I can't recreate the same VM?

config of working VM:

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;ide2;net0;ide0
cores: 12
cpu: host
cpuunits: 5000
efidisk0: local-lvm:vm-400-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:07:00,pcie=1,x-vga=1
ide0: none,media=cdrom
ide2: none,media=cdrom
machine: pc-q35-7.0
memory: 16384
meta: creation-qemu=7.0.0,ctime=1665367325
name: Gamestation
net0: virtio=26:CE:B2:DF:03:55,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
sata0: local-lvm:vm-400-disk-1,size=220G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=ba498e7d-44eb-4735-b0f5-bbf3f86c2681
sockets: 1
tpmstate0: local-lvm:vm-400-disk-2,size=4M,version=v2.0
vga: none

config of new VM:

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;scsi0
cores: 6
cpu: host
cpuunits: 5000
efidisk0: local-lvm:vm-136-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:07:00,pcie=1,x-vga=1
machine: pc-q35-7.0
memory: 8192
meta: creation-qemu=7.2.10,ctime=1719452367
name: win11
net0: virtio=02:15:E6:D0:1E:44,bridge=vmbr0,firewall=1
numa: 0
onboot: 0
ostype: win11
sata0: local:iso/Win11_English_x64.iso,media=cdrom,size=5434622K
sata1: local:iso/virtio-win.iso,media=cdrom,size=519030K
scsi0: local-lvm:vm-136-disk-1,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=d6c87c31-c099-0d06-2e28-c537b17c082d
sockets: 1
tpmstate0: local-lvm:vm-136-disk-2,size=4M,version=v2.0
vga: none

 

[gfngfn256]

scsi0: local-lvm:vm-136-disk-1,discard=on,size=40G,ssd=1

Not sure what your exact problem is - but things I'd start with.

1. Change this main HD (C of the Windows 11 VM to a SATA bus (like the working VM).
2. Change its size to 64GB or greater; since this is the minimum Windows 11 requirement AFAIK (your working VM one is 220GB).

 

[leesteken]

I see this message in the noVNC console: "guest has not initialized the display (yet)".

vga: none

I would fully expect such an error message because the virtual Display is set to None.

I don't understand why noVNC would work on your old VM. I would have thought you needed a physical display connected to it so see any output.

 

[remz1337]

I forgot to mention this detail. Before adding the GPU (and setting the display to none), I'm using the default display. I can install and use Windows 11 using the default display and machine type i440fx. See config and screenshot below:

working config of new VM (before adding PCIe GPU):

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;scsi0
cores: 6
cpu: host
cpuunits: 5000
efidisk0: local-lvm:vm-136-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-i440fx-7.0
memory: 8192
meta: creation-qemu=7.2.10,ctime=1719452367
name: win11
net0: virtio=02:15:E6:D0:1E:44,bridge=vmbr0,firewall=1
numa: 0
onboot: 0
ostype: win11
sata0: local:iso/Win11_English_x64.iso,media=cdrom,size=5434622K
sata1: local:iso/virtio-win.iso,media=cdrom,size=519030K
scsi0: local-lvm:vm-136-disk-1,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=d6c87c31-c099-0d06-2e28-c537b17c082d
sockets: 1
tpmstate0: local-lvm:vm-136-disk-2,size=4M,version=v2.0

And as soon as I switch to q35 machine, I get no display (Guest has not initialized the display (yet). See config and screenshot below:

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;scsi0
cores: 6
cpu: host
cpuunits: 5000
efidisk0: local-lvm:vm-136-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-7.0
memory: 8192
meta: creation-qemu=7.2.10,ctime=1719452367
name: win11
net0: virtio=02:15:E6:D0:1E:44,bridge=vmbr0,firewall=1
numa: 0
onboot: 0
ostype: win11
sata0: local:iso/Win11_English_x64.iso,media=cdrom,size=5434622K
sata1: local:iso/virtio-win.iso,media=cdrom,size=519030K
scsi0: local-lvm:vm-136-disk-1,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=d6c87c31-c099-0d06-2e28-c537b17c082d
sockets: 1
tpmstate0: local-lvm:vm-136-disk-2,size=4M,version=v2.0

 

[leesteken]

I forgot to mention this detail. Before adding the GPU (and setting the display to none), I'm using the default display. I can install and use Windows 11 using the default display and machine type i440fx.

And as soon as I switch to q35 machine, I get no display (Guest has not initialized the display (yet). See config and screenshot below:

Windows typically does not like you changing the (virtual) motherboard (or BIOS/UEFI) after installation. Try install Windows again with q35 from the start.

 

[remz1337]

You got me thinking. I did a few more tests. I created a VM through the web UI with the same config and this one boots properly in the noVNC console, while the one I create using the qm create command does not...

Here's the config of the VM created through the web UI (works):

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;scsi0
cores: 6
cpu: host
efidisk0: local-lvm:vm-137-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-7.2
memory: 8192
meta: creation-qemu=7.2.10,ctime=1719778133
name: test
net0: virtio=72:CB:AC:B5:20:CD,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
sata0: local:iso/Win11_English_x64.iso,media=cdrom,size=5434622K
sata1: local:iso/virtio-win.iso,media=cdrom,size=519030K
scsi0: local-lvm:vm-137-disk-1,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=80b37c54-b349-4763-86ad-d2c03f258bc1
sockets: 1
tpmstate0: local-lvm:vm-137-disk-2,size=4M,version=v2.0
vmgenid: eb694a77-dd46-4856-965b-a277d9662036

Here's the qm create command that I used with the resulting config (doesn't work):

pvesm alloc local-lvm 139 vm-139-disk-0 4M
pvesm alloc local-lvm 139 vm-139-disk-1 40G
pvesm alloc local-lvm 139 vm-139-disk-2 4M
qm create 139 -agent 1 -machine q35 -onboot 0 -bios ovmf -cpu host -cores 6 -balloon 4096 -memory 8192 -name win11 -net0 virtio,bridge=vmbr0,macaddr=02:7C:82:64:D1:DE,firewall=1 -ostype win11 -scsihw virtio-scsi-pci -efidisk0 local-lvm:vm-139-disk-0,efitype=4m,pre-enrolled-keys=1   -scsi0 local-lvm:vm-139-disk-1,discard=on,ssd=1,size=40G   -tpmstate0 local-lvm:vm-139-disk-2,size=4M,version=v2.0   -sata0 /var/lib/vz/template/iso/Win11_English_x64.iso,media=cdrom   -sata1 /var/lib/vz/template/iso/virtio-win.iso,media=cdrom   -smbios1 uuid=447cbfda-2713-52f9-fb6e-3107d0db3126   -boot order=sata0;scsi0

agent: 1
balloon: 4096
bios: ovmf
boot: order=sata0;scsi0
cores: 6
cpu: host
efidisk0: local-lvm:vm-139-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-7.2
memory: 8192
meta: creation-qemu=7.2.10,ctime=1719779014
name: win11
net0: virtio=02:7C:82:64:D1:DE,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
sata0: local:iso/Win11_English_x64.iso,media=cdrom,size=5434622K
sata1: local:iso/virtio-win.iso,media=cdrom,size=519030K
scsi0: local-lvm:vm-139-disk-1,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=447cbfda-2713-52f9-fb6e-3107d0db3126
sockets: 1
tpmstate0: local-lvm:vm-139-disk-2,size=4M,version=v2.0
vmgenid: 1eca56d9-11b4-4506-b70f-d148e882efab

 

[gfngfn256]

Here's the qm create command that I used with the resulting config (doesn't work):

Now you've got me stumped! The 2 VM's appear identical (bar the smbios1 uuid you created yourself - but I would imagine shouldn't affect anything).

Maybe try setting-up a new VM without pre-allocating disks with pvesm alloc , but with the inline option in qm create using the special syntax STORAGE_ID:SIZE_IN_GiB as described in the docs. (My thinking: the pvesm alloc somehow misconfigures the correct format for local-lvm).

 

[remz1337]

Thanks for the hint. Indeed there seems to be an issue with pvesm alloc. I tried using the allocated discs with import-from but same issue. What solved was your suggestion to let qm create allocate the disks. This is the final command that worked (notice the 1G for EFI and TPM that get overwritten to 4M)

qm create 136 -agent 1 -machine q35 -onboot 0 -bios ovmf -cpu host -cores 6 -balloon 4096 -memory 8192   -name win11 -net0 virtio,bridge=vmbr0,macaddr=02:F0:6B:4C:8A:AF,firewall=1 -ostype win11 -scsihw virtio-scsi-single  -efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=1   -scsi0 local-lvm:40,discard=on,ssd=1,   -tpmstate0 local-lvm:1,version=v2.0   -sata0 /var/lib/vz/template/iso/Win11_English_x64.iso,media=cdrom   -sata1 /var/lib/vz/template/iso/virtio-win.iso,media=cdrom   -smbios1 uuid=44572ab2-a8be-b959-d9e9-9afefb5f1ef0   -boot order=sata0;scsi0

Should I open a bug report or is that intended behaviour?

 

[gfngfn256]

Should I open a bug report or is that intended behaviour?

Yes this is the intended behaviour as per the docs I linked in my above post.

However you may have discovered what's wrong when you use the pre-allocation method. What I'm thinking is that the system allocates the sizes you have given, but later does not adjust the sizes of the EFI & TPM disks resulting in an unbootable system.

To test my theory, pre-allocate only the main Windows system drive but put the EFI &TPM disks in the qm create one. If it boots, we've both learned something.

 

[remz1337]

I have confirmed your theory. What I don't understand is why pre-allocating the EFI and TPM disks does not work with q35 machine but works with i440fx machine.

 

[gfngfn256]

I have confirmed your theory

Good.

What I don't understand is why pre-allocating the EFI and TPM disks does not work with q35 machine but works with i440fx machine.

Most likely; the 2 machines incorporate EFI & TPM in a different manner, & the i440fx/chipset does not require a specific EFI & TPM (disk) size.


Thanks for the lessons we have learned together. I believe your issue is resolved.
Maybe tag prefix the thread-title with [SOLVED], (upper right hand corner under title).