GPU pass-through no longer functioning since upgrade

[Charlie B]

since upgrading my GPU pass-throughs no longer function. the virtual machines still see the video card and allow me to reinstall the video driver however they will not start the video card.

agent: 1
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: store13-a:vm-2222-disk-2,size=4M
hostpci0: de:00,pcie=1,x-vga=on
ide2: none,media=cdrom
machine: q35
memory: 98304
name: rsmsngpu222
net0: virtio=FA:08:FC:74:A3:E3,bridge=vmbr320
numa: 0
ostype: win10
scsi0: store13-a:vm-2222-disk-1,backup=0,discard=on,replicate=0,size=200G
scsi1: store13-a:vm-2222-disk-3,backup=0,discard=on,replicate=0,size=1536G
scsihw: virtio-scsi-pci
smbios1: uuid=6b63181b-2497-4b46-8b5e-ca8ab42548b9
sockets: 4
vga: none

Virtual Environment 5.4-6
proxmox-ve: 5.4-1 (running kernel: 4.15.18-15-pve)
pve-manager: 5.4-6 (running version: 5.4-6/aa7856c5)
pve-kernel-4.15: 5.4-3
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.15-1-pve: 4.15.15-6
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph: 12.2.12-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-10
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-52
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-43
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-39
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-2
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-52
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[dcsapak]

the virtual machines still see the video card and allow me to reinstall the video driver however they will not start the video card.

how do you install the video driver if you have no output ? how do you connect to the vm ?
how was it working before? from which version did you upgrade?

 

[Charlie B]

it's using the generic Windows basic driver. I normally connect to the VM through VNC installed on the VM itself.
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36

we are using Nvidia P5000 cards. this happened a crossed six different hosts. they are all exhibiting the same behavior.

 

[dcsapak]

anything in the log/journal/dmesg of the host or the guest ?