[SOLVED] Windows 10 VM GPU Passthrough stuck on boot screen

N

nekodesew

New Member
Oct 17, 2021
22
1
3
31
Hello,

My windows 10 VM gets stuck on this screen. I thought I did everything correctly, but it looks like I missed something.
1647405696356.png

Parts:
  • Ryzen 7 1700
  • Aorus B450M motherboard
  • 32 GB ram
  • 240 SSD GB, 2*1TB (Raid 1), 2*4TB (Raid 1)
  • RTX 3060 (planning to use for windows)
As of now I am able to boot into my Windows 10 VM no problem when I remove my gpu. Some arguments in my config file was so I can use Hyper V but I forgot which exactly. If anyone could help out that would be great. I don't know what else I could try, been stuck on this issue for a few days.

Code: 
root@pve:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.15.27-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-13
pve-kernel-helper: 7.1-13
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.27-1-pve: 5.15.27-1
pve-kernel-5.15.19-2-pve: 5.15.19-3
pve-kernel-5.13.19-6-pve: 5.13.19-14
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libqb0: not correctly installed
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-7
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-6
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Code: 
./killvm.sh 100
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off"
balloon: 8192
bios: ovmf
boot: order=virtio0;ide2
cores: 10
cpu: host,hidden=1,flags=+pcid
efidisk0: HDD4TB:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
ide2: local:iso/Windows10.iso,media=cdrom
machine: pc-q35-6.1
memory: 20480
meta: creation-qemu=6.1.0,ctime=1647243943
name: WIN10
net0: virtio=4A:56:23:2B:7B:F0,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=9ce8f82e-ca4d-4fb2-94f6-4b60633e535b
sockets: 1
tablet: 0
virtio0: HDD4TB:vm-100-disk-0,cache=writeback,size=3000G
vmgenid: 4a11fb45-a624-4cce-ba04-dfa06516a619

Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt"
GRUB_CMDLINE_LINUX=""

Code: 
lspci -v
06:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 3657
        Flags: fast devsel, IRQ 255, IOMMU group 13
        Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
        Memory at b0000000 (64-bit, prefetchable) [size=256M]
        Memory at c0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at fc000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
        Capabilities: [d00] Lane Margining at the Receiver <?>
        Capabilities: [e00] Data Link Feature <?>
        Kernel modules: nvidiafb, nouveau

Code: 
root@pve:~# lspci -n -s 06:00
06:00.0 0300: 10de:2503 (rev a1)
06:00.1 0403: 10de:228e (rev a1)

Code: 
/etc/modprobe.d/vfio.conf 
options vfio-pci ids=10de:2503,10de:228e disable_vga=1
 
hi,

have you also blacklisted the nvidia drivers from the host as described in [0] ?

in our guide it's also described to set the machine parameter to q35 (you can check if that makes any difference for you)

nekodesew said:
As of now I am able to boot into my Windows 10 VM no problem when I remove my gpu.
Click to expand...
also i don't see a hostpci line in your VM configuration, so is this the working configuration that you posted?
can you show us the config with the passthrough as well?

nekodesew said:
Some arguments in my config file was so I can use Hyper V but I forgot which exactly.
Click to expand...
should be the args line, you could comment that for the time being to make it simpler to debug.

you should also check if you GPU is compatible with UEFI boot (see the corresponding section on the link with the rom parser)

hope this helps!

[0]: https://pve.proxmox.com/wiki/Pci_passthrough#GPU_Passthrough
 
have you also blacklisted the nvidia drivers from the host as described in [0] ?
Click to expand...
Yes, I have the following drivers in my /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

also i don't see a hostpci line in your VM configuration, so is this the working configuration that you posted?
can you show us the config with the passthrough as well?
Click to expand...
Code: 
#./killvm.sh 100
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off"
balloon: 8192
bios: ovmf
boot: order=virtio0;ide2
cores: 10
cpu: host,hidden=1,flags=+pcid
efidisk0: HDD4TB:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:06:00,pcie=1
ide0: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
ide2: local:iso/Windows10.iso,media=cdrom
machine: pc-q35-6.1
memory: 20480
meta: creation-qemu=6.1.0,ctime=1647243943
name: WIN10
net0: virtio=4A:56:23:2B:7B:F0,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=9ce8f82e-ca4d-4fb2-94f6-4b60633e535b
sockets: 1
tablet: 0
virtio0: HDD4TB:vm-100-disk-0,cache=writeback,size=3000G
vmgenid: 4a11fb45-a624-4cce-ba04-dfa06516a619

you should also check if you GPU is compatible with UEFI boot (see the corresponding section on the link with the rom parser)
Click to expand...
just to confirm i need to run
git clone https://github.com/awilliam/rom-parser

in proxmox shell terminal?
 
Code: 
-bash: git: command not found
means it's not installed, try installing the "git" package first: apt install git ;)

nekodesew said:
i added x-vga=on in my config file but i dont think its working. the ip address isnt showing after 5 minutes.
Click to expand...
what do you mean? is it booting normally now, and not getting an IP? please be more precise
 
Windows will use your gpu-card to display when you have installed drivers in Windows.
enable remote access before gpu-passthrough,and you can confirm whether windows is running。
 
oguz said:
try apt install make


does it work with display set to SPICE?
Click to expand...
SPICE does not work. It turns on but gets stuck at the boot screen (where the dots move in a circle shape). I did try to use Standard VGA as a display and that booted but it prompted me to a recovery screen.

I tried the code again and this is what happened:
root@pve:~# git clone https://github.com/awilliam/rom-parser
fatal: destination path 'rom-parser' already exists and is not an empty directory.
root@pve:~# cd rom-parser
root@pve:~/rom-parser# make
gcc -o rom-parser rom-parser.c
make: gcc: Command not found
make: *** [Makefile:4: rom-parser] Error 127
root@pve:~/rom-parser#
 
nekodesew said:
make: gcc: Command not found
Click to expand...

you also need to install gcc, the same way as the above missing commands :)

nekodesew said:
SPICE does not work. It turns on but gets stuck at the boot screen (where the dots move in a circle shape). I did try to use Standard VGA as a display and that booted but it prompted me to a recovery screen.
Click to expand...
okay. have you removed the arg flags for hyper-v? does it work without those?

it would make sense to get back to a plain working config, take a snapshot or backup and then start changing it from there.
 
you also need to install gcc, the same way as the above missing commands :)
Click to expand...
thats what I thought
Code: 
root@pve:~# cd rom-parser
root@pve:~/rom-parser# make
gcc -o rom-parser rom-parser.c
rom-parser.c:1:10: fatal error: fcntl.h: No such file or directory
 #include <fcntl.h>
          ^~~~~~~~~
compilation terminated.
make: *** [Makefile:4: rom-parser] Error 1
root@pve:~/rom-parser#

i removed my args line and i got to the screen below. this is the same screen i got when i was changing my display settings. a friendly reminder that my vm boots fine if i remove my gpu from my vm.
Screen Shot 2022-03-17 at 6.17.42 AM.png

for reference i used these two guides:
https://dannyda.com/2021/01/01/how-...walkthrough-means-we-assume-everything-works/

https://dannyda.com/2020/09/21/how-to-enable-nested-hypervisor-virtualization-on-proxmox-ve-pve/
 
in case someone has this issue in the future

i also attached my gpu and had to reinstall windows. im surprised it worked.

this is my config file:
Code: 
#./killvm.sh 100
agent: 1
balloon: 8192
bios: ovmf
boot: order=virtio0;ide0;ide2
cores: 10
cpu: host,hidden=1,flags=+virt-ssbd;+amd-ssbd
efidisk0: HDD4TB:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:06:00,pcie=1,x-vga=1
ide0: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
ide2: local:iso/Windows10.iso,media=cdrom
machine: pc-q35-6.1
memory: 20480
meta: creation-qemu=6.1.0,ctime=1647243943
name: WIN10
net0: virtio=4A:56:23:2B:7B:F0,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=9ce8f82e-ca4d-4fb2-94f6-4b60633e535b
sockets: 1
tablet: 0
vga: std
virtio0: HDD4TB:vm-100-disk-0,cache=writeback,size=3000G
vmgenid: 4a11fb45-a624-4cce-ba04-dfa06516a619
args: -cpu 'host,+svm,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off,hypervisor=off'

edit: seems like its working.
pros: hyper v is working and gpu is split among 3 vms.
cons: as expected hyper-v vms isnt not as smooth as i liked. each of the 3 vms has 4 cores and 8gb ram. 1 vm has 50% of the 3060 and the other two has 25% each. the slowness could be a number of factors with the equipment i have at home.
all and all, i consider this a win.
 
Last edited:
  • Like
Reactions: oguz
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom