Display Type VirGL GPU (virtio-gl) Issues w/ Suspend & Snapshots

[crfslick]

Linux PVE guests with display types set to "virtio-gl" don't resume properly from hibernation and snapshots. Resuming results in no display output and requires resetting the guest to fix.

The the only sign that something has gone wrong is in the task output (no errors in dmesg).

Resuming suspended VM
activating and using 'local-storage:vm-212-state-suspend-2023-11-23' as vmstate
kvm: Failed to load virtio-gpu:virtio-gpu
kvm: error while loading state for instance 0x0 of device '0000:00:01.0/virtio-gpu'
kvm: Error while loading VM state: Invalid argument
Resumed VM, removing state
TASK OK

For context, the PVE host is running an AMD Ryzen 5000 series "G" skew cpu. Issue is present with both SeaBIOS and OVMF machine configurations. Required libraries "libgl1" and "libegl1" are installed, as per the documentation, and virtio-gl generally works fine otherwise.

Is anyone else experiencing this issue?

Edit: host is running PVE 8.1.3, kernel 6.5.11-4-pve

 

[LnxBil]

Yes, sounds reasonable. You're breaking the strict virtualization by using hardware (virtio-gpu), which has not yet a snapshotable or hibernatable state. Maybe it'll be in the future, but this is far from trivial and may lead (like you discovered) to some problems. Easiest way is to snapshot in offline mode and don't hibernate - like with everything else. Online snapshot is always risky.

 

[crfslick]

Yes, sounds reasonable. You're breaking the strict virtualization by using hardware (virtio-gpu), which has not yet a snapshotable or hibernatable state. Maybe it'll be in the future, but this is far from trivial and may lead (like you discovered) to some problems. Easiest way is to snapshot in offline mode and don't hibernate - like with everything else. Online snapshot is always risky.

Makes sense. I guess I didn't realize how close virtio-gpu was to actual hardware pass-through. Thanks

Although it would be nice if the GUI informed the user of this when hibernating / making a snapshot of a system with virtio-gpu enabled. As it is, the resume / restore task "succeeds" (TASK OK) but fails silently with errors that are only found in the task log which isn't shown by default.

 

[LnxBil]

Although it would be nice if the GUI informed the user of this when hibernating / making a snapshot of a system with virtio-gpu enabled. As it is, the resume / restore task "succeeds" (TASK OK) but fails silently with errors that are only found in the task log which isn't shown by default.

Agreed.

 

[fiona]

Hi,
can you please share the configuration of an affected guest qm config <ID> and output of pveversion -v. Usually QEMU complains up front about migration blockers (snapshots/hibernation use the same logic/code under the hood), it's a bit strange it doesn't consider this to be one.

 

[crfslick]

Hi,
can you please share the configuration of an affected guest qm config <ID> and output of pveversion -v. Usually QEMU complains up front about migration blockers (snapshots/hibernation use the same logic/code under the hood), it's a bit strange it doesn't consider this to be one.

agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: x86-64-v2-AES
efidisk0: local-storage:vm-212-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: remote-storage:iso/ubuntu-20.04.3-desktop-amd64.iso,media=cdrom,size=2999936K
machine: q35
memory: 8192
meta: creation-qemu=8.0.2,ctime=1700759230
name: test
net0: virtio=22:75:83:8D:F7:82,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-storage:vm-212-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=17d017d9-1dd2-4e2a-adf0-a302e56e6d78
sockets: 1
vga: virtio-gl
vmgenid: b33af661-f907-44d8-918e-f498728accba

proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.0.9
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
proxmox-kernel-6.5: 6.5.11-4
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.4
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.9
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-1
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.2
pve-qemu-kvm: 8.1.2-4
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3