[SOLVED] Reinstalled Proxmox and cannot get PCIE passthrough of GPU working again.

[jtracy]

So I previously had GPU passthrough working to a windows 10 VM. But I had had a harddrive failure on my proxmox server that forced me to reinstall. I have been following every tutorial and mostly rememeber what I had done previously but and having issue with it working for Windows 10. I can start up a Ubuntu Live CD and it will passthrough just fine (Well I don't see and BIOS information but it eventually intializes that cards and works just fine). When I install WIndows 10 I can get to the point of installing the NVidia driver and it says it is working just fine but nothing shows up. If I reboot then I start to get code 43 errors.

Here is what I have done so far.

New Proxmox install using UEFI to a RAID Z1 on two SSDs. The server is a AMD Ryzen 7 3700X on a ASRock X570M with 96G of ram. The GPU is a EVGA 1070 installed in the top slot for the GPU. There are 2 Samsung 860 256G ssds for the install, a Samsung 970 512G NVME drive and WD 1TB NVME drive.

root@pve01:~# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt vfio-pci.ids=10de:1b81,10de:10f0,1022:149c,1022:1487 vfio-pci.disable_vga=1 video=efifb:off

root@pve01:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

root@pve01:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

root@pve01:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

root@pve01:~# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0

I have also dumped the rom of the GPU using GPU-Z (I had done this last time I got it working). But it seems to not make a difference with the current install and the Ubuntu version worked without the rom.

root@pve01:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.4-4 (running version: 6.4-4/337d6701)
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-1
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[dcsapak]

can you post the vm config (qm config ID) and the output of 'dmesg' ?

 

[jtracy]

Thanks for getting back to me. I posted dmesgs of each start of the VM and also attached a full dmesg from after a reboot, then staring/stoping the Ubuntu VM and then starting the Windows 10 VM.

New Windows not working:

root@pve01:/etc/pve/qemu-server# cat 111.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 12
cpu: host
efidisk0: BigNFast-VMs:vm-111-disk-1,size=4M
hostpci0: 09:00,pcie=1,x-vga=1
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide0: local:iso/Nvidia.iso,media=cdrom,size=685778K
ide2: none,media=cdrom
machine: pc-q35-5.2
memory: 8192
name: Win10
net0: virtio=5E:FD:56:44:A1:A0,bridge=vmbr2,firewall=1
numa: 0
ostype: win10
parent: Installed_Updated
scsi0: BigNFast-VMs:vm-111-disk-0,discard=on,size=400G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=17decb17-bc7b-4fe3-a5c6-96d13a928793
sockets: 1
vga: std
vmgenid: bd900b9e-a27d-4920-9bc5-6eef0d6c97f5

[Installed_Updated]
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 12
cpu: host
efidisk0: BigNFast-VMs:vm-111-disk-1,size=4M
hostpci0: 09:00,pcie=1,x-vga=1
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide0: local:iso/virtio-win-0.1.185.iso,media=cdrom,size=402812K
ide2: local:iso/Windows10.iso,media=cdrom
machine: pc-q35-5.2
memory: 8192
name: Win10
net0: virtio=5E:FD:56:44:A1:A0,bridge=vmbr2,firewall=1
numa: 0
ostype: win10
scsi0: BigNFast-VMs:vm-111-disk-0,discard=on,size=400G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=17decb17-bc7b-4fe3-a5c6-96d13a928793
snaptime: 1620252741
sockets: 1
vga: std
vmgenid: bd900b9e-a27d-4920-9bc5-6eef0d6c97f5

root@pve01:/etc/pve/qemu-server# dmesg -c
root@pve01:/etc/pve/qemu-server# qm start 111
root@pve01:/etc/pve/qemu-server# dmesg
[56404.889076] device tap111i0 entered promiscuous mode
[56404.904606] fwbr111i0: port 1(fwln111i0) entered blocking state
[56404.904611] fwbr111i0: port 1(fwln111i0) entered disabled state
[56404.904658] device fwln111i0 entered promiscuous mode
[56404.904685] fwbr111i0: port 1(fwln111i0) entered blocking state
[56404.904687] fwbr111i0: port 1(fwln111i0) entered forwarding state
[56404.906799] vmbr2: port 2(fwpr111p0) entered blocking state
[56404.906802] vmbr2: port 2(fwpr111p0) entered disabled state
[56404.906842] device fwpr111p0 entered promiscuous mode
[56404.906868] vmbr2: port 2(fwpr111p0) entered blocking state
[56404.906870] vmbr2: port 2(fwpr111p0) entered forwarding state
[56404.908944] fwbr111i0: port 2(tap111i0) entered blocking state
[56404.908948] fwbr111i0: port 2(tap111i0) entered disabled state
[56404.909008] fwbr111i0: port 2(tap111i0) entered blocking state
[56404.909010] fwbr111i0: port 2(tap111i0) entered forwarding state
[56405.853485] vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[56405.854756] vfio-pci 0000:09:00.0: No more image in the PCI ROM
[56405.906388] vfio-pci 0000:06:00.3: enabling device (0000 -> 0002)
[56408.526171] vfio-pci 0000:09:00.0: No more image in the PCI ROM
[56408.526191] vfio-pci 0000:09:00.0: No more image in the PCI ROM

New Ubuntu Live Working:

root@pve01:/etc/pve/qemu-server# cat 112.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
efidisk0: BigNFast-VMs:vm-112-disk-1,size=4M
hostpci0: 09:00,pcie=1,x-vga=1
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide2: local:iso/ubuntu-21.04-desktop-amd64.iso,media=cdrom
machine: q35
memory: 8192
name: Ubuntu
net0: virtio=82:2D:98:5E:07:6C,bridge=vmbr2,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=85083761-014e-4c01-bbbf-f93958a3022f
sockets: 1
vga: none
virtio0: BigNFast-VMs:vm-112-disk-0,size=32G
vmgenid: c17d3b7d-c1a1-4594-9d66-80ee1c53d351

root@pve01:/etc/pve/qemu-server# dmesg -c

root@pve01:/etc/pve/qemu-server# qm start 112

root@pve01:/etc/pve/qemu-server# dmesg
[56237.361727] device tap112i0 entered promiscuous mode
[56237.378133] fwbr112i0: port 1(fwln112i0) entered blocking state
[56237.378137] fwbr112i0: port 1(fwln112i0) entered disabled state
[56237.378179] device fwln112i0 entered promiscuous mode
[56237.378209] fwbr112i0: port 1(fwln112i0) entered blocking state
[56237.378210] fwbr112i0: port 1(fwln112i0) entered forwarding state
[56237.380718] vmbr2: port 2(fwpr112p0) entered blocking state
[56237.380722] vmbr2: port 2(fwpr112p0) entered disabled state
[56237.380768] device fwpr112p0 entered promiscuous mode
[56237.380793] vmbr2: port 2(fwpr112p0) entered blocking state
[56237.380795] vmbr2: port 2(fwpr112p0) entered forwarding state
[56237.382364] fwbr112i0: port 2(tap112i0) entered blocking state
[56237.382367] fwbr112i0: port 2(tap112i0) entered disabled state
[56237.382421] fwbr112i0: port 2(tap112i0) entered blocking state
[56237.382422] fwbr112i0: port 2(tap112i0) entered forwarding state
[56238.310654] vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[56238.312101] vfio-pci 0000:09:00.0: No more image in the PCI ROM
[56238.362328] vfio-pci 0000:06:00.3: enabling device (0000 -> 0002)
[56240.934895] vfio-pci 0000:09:00.0: No more image in the PCI ROM
[56240.934942] vfio-pci 0000:09:00.0: No more image in the PCI ROM

Old Windows that used to work:

agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 12
efidisk0: Fast-VMs:vm-100-disk-0,size=4M
hostpci0: 09:00,pcie=1,x-vga=1,romfile=GP104.rom
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide2: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 12288
name: Windows
net0: virtio=4E:BF:4A:0A:B9:06,bridge=vmbr2,firewall=1
net1: virtio=66:5B:C9:26:69:C7,bridge=vmbr3,firewall=1
numa: 0
onboot: 1
ostype: win10
protection: 1
rng0: source=/dev/urandom
scsi0: BigNFast-VMs:vm-100-disk-0,discard=on,size=400G
scsi1: Fast-VMs:vm-100-disk-2,size=1G
scsihw: virtio-scsi-pci
smbios1: uuid=e2d2f136-6c4f-44e3-b12a-2cfe0128e8ba
sockets: 1
startup: order=1,up=15
unused0: NAS00:100/vm-100-disk-0.qcow2
unused1: NAS00:100/vm-100-disk-1.qcow2
unused2: Fast-VMs:vm-100-disk-1
vga: qxl
vmgenid: 3fcc234c-45a8-4823-878d-2a9d1b36316e

 

[dcsapak]

ok, maybe it's the fact that you used to have the 'qxl' vga (SPICE in the gui) and now you have 'vga' ?
can you try to change it, or set 'none' at all?

 

[jtracy]

Switched it to none and nothing is happing on the GPU.

 

[jtracy]

I had also asked for help on the VIFO discord and they noticed I did not strip the headers out of the GPU bios rom file. Once I did that I can passthrough the GPU to Windows 10. But it only works after the OS has started loading drivers. Is there a way to get the OVMF bios to work correctly when starting the VM?