Virtio-gpu broken after upgrade pve7.4 to pve8

[tanyastawicki]

Halo,

After ( successfull ) upgrading our ProxMox PVE ( bulleye - bookworm, pve7 to pve8 ) the console of VMs, using the Display type VirtIO-gpu stopped working. When right-clicking the VM and choosing > console not the console opens, but a standard Save AS Windows with the file pve-spice.vv appears on screen.

Anybody experience the same error? Anybody knows how to fix it?

Thanks in advance for any reply

Tanya

 

[gfngfn256]

Your getting a file to connect to SPICE, you'll need a SPICE client to be able to connect.

Why your VirtIO-GPU stopped functioning has to be addressed. Possibly the VirtIO drivers need updating in the VM.

Maybe share your VMID.conf file for investigation. What does Display type for VM show in GUI.

 

[tanyastawicki]

Helo GFNG

The display shows: "VirtIO-GPU (virtio)"

here my VMID.conf The VM is one of more VMs running running ChromeOS flex. We use them for Chromebook testing.
It used to work flawlessly on PVE 7.1 til 7.4.

balloon: 0
boot: order=sata0
cores: 2
machine: q35
memory: 4096
meta: creation-qemu=6.1.0,ctime=1666693440
name: ChromeOS
net0: virtio=42:E8:F8E:AF:2F,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
sata0: local-zfs:vm-121-disk-0,cache=unsafe,discard=on,size=16G,ssd=1
smbios1: uuid=aa3f39c6-6852-46d6-9e20-134ca7232c74
sockets: 1
vga: virtio
vmgenid: 97fa8e48-dcce-49a8-807b-8e9319299d0c

 

[gfngfn256]

This may be linked to your problem.
Can you in fact use SPICE. If possible, I would use it, as in general its usually superior as a remote viewer.
If not, maybe try VirGL-GPU, & see if that works.

 

[tanyastawicki]

problem partly solved ...
dont start the console as usual: right clicking ther VM but start it in the right upper corner with the down arrow on the "console"-icon



There we go: a "normal" noVNC console appears.

But now we see an even bigger problem with pve8: ChromeOS Flex crashes while starting, reboots, crashes while starting enz... A boot loop.

We try to setup a new VM ChromeOS Flex: same crashes
We try to export a working ChromeOS from 7.1 and import in our upgraded PVE: same crashes.
We try some different hardware setup ( bios q35 vs 440xf ) crashes

Sometimes: after 8 or 9 boot loops a VM succesfull starts ChromeOS !!! but on PVE7 the virtual Chromebooks never ever crashed.

I guess PVE8 is not for us. Luckly most of our nodes are still on 7

Are there people out there who have FlexOS stable running on PVE 8.x ?

 

[gfngfn256]

We try some different hardware setup ( bios q35 vs 440xf ) crashes

Have you tried changing the Hardware, Processors, Type in the VM? What is it currently set on?

 

[tanyastawicki]

No I'll give it a try !

It is now set on "host" 1 socket, 2 cores,
( the host processor on this host is AMD ryzen 7 2000 series )

 

[gfngfn256]

Maybe start with x86-64-v2-AES

 

[tanyastawicki]

I have tried different CPU setups

x86-64-v2-AES
x86-64-v2
x86-64-v3
kvm64
qemu64
some haswells

on two different PVE8 hosts. This one:


and this one:

CPU(s) 40 x Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (2 Sockets)

Kernel Version Linux 6.5.11-4-pve (2023-11-20T10:19Z)

Boot Mode Legacy BIOS

Manager Version pve-manager/8.1.3/b46aac3b42da5d15

but both PVE 8 machines suffer from the Chrome0s Flex crashes while the same ( migrated ) Chrome VMs boot always correct on al our PVE7 hosts. ( both Xeon and Ryzen )


Other question: Is there a way to rollback from PVE 8.x to PVE 7 ?

 

[Ramalama]

you could try to boot into an older Kernel. Or install for example pve-kernel-6.2 then boot into it to test if it resolves.
A Downgrade is possible, but its not simple and possibly even break things.
Basically you'll need to check apt history and install all the old packages.

There are apt alternatives like "Nala", which makes rollbacks possible "the easy way" im just not sure if Nala can Rollback Updates from APT.
Usually if you Upgrade with Nala, nala logs all the changed files and so on, so you can revert the upgrade later on. However, you can try it, but i wouldn't recommend generally a Downgrade.

Cheers

 

[gfngfn256]

The clean & working way, would be to use an old ISO of a known PVE version (from here) that works with your VM & reinstall fresh.

If you want to be adventurous, you could try installing the older working PVE (with above ISO) as a VM inside the newer one, i.e.: nested virtualization. Then put a Chrome Flex OS VM inside that one. This is obviously for development only.

 

[tanyastawicki]

you could try to boot into an older Kernel.
Oke.. that is an idea
As I mentioned: I upgraded the proxmox server, so maybe in the Grub start menu there is a possibility boot from an older kernel. I don't know: We don't have a monitor attached to the Proxmox server.

we indeed have these kernels in /boot :

[ATTACH type="full"]67061[/ATTACH]


must wait for the right time to force a reboot with monitor and keyboard..
Keep you informed....

Still wanna know of others have more success with ChromeOS Flex en PVE 8. I'm still hoping for a breakthrough....

 

[tanyastawicki]

I finally was able to reboot the Proxmox server with monitor and keyboard...

I indeed have a "grub-choice" to boot in kernel:
1) 5.15.149-1-pve
2) 6.8.4-2-pve

when I choose the "5"-one the server boot successfully in kernel 5.15 , the server is reachable and working. But the crashes of ChromeOS Flex are there to stay :-( . The PVE is still 8.2.2 as with the newer kernel .

Rolling back the PVE to 7 is apparently not that easy....

No one out there with a successfully applied ChromeOS flex on PVE 8.x ??? ( https://i12bretro.github.io/tutorials/0718.html )

 

[gfngfn256]

I would do a clean install with an old ISO - as I mentioned above.

 

[tanyastawicki]

I tried to get Chromeos Flex running on a Kali Linux and on a Windows 11 device, with on both qemu installed. The frustating thing is that it successfully runs on both devices.

command line:

qemu-system-x86_64.exe -drive format=qcow2,file=g:\linux2020.qcow2 -m 4G -display "gtk,gl=on,show-cursor=on" -device virtio-vga-gl -usb -device usb-tablet -smp cores=4

and is runs without crashes.. with the latest versions of qemu
PS C:\msys64\mingw64\bin> .\qemu-system-x86_64.exe --version
QEMU emulator version 9.0.0

 

[ArtanisKrall]

Hi, I'm having the same issue and found nothing else in the Internet. Did you manage to solve the problem?
For me it installed correctly, but it boot loops after installation. I've tried other types of Display (VGA, Vmware, SPICE) without succes.

 

[gfngfn256]

I tried to get Chromeos Flex running on a Kali Linux and on a Windows 11 device, with on both qemu installed. The frustating thing is that it successfully runs on both devices.

So how about installing a Kali Linux (or similar) VM in PVE & try running within it a qemu of ChromeOS?

 

[tanyastawicki]

Hi, I'm having the same issue and found nothing else in the Internet. Did you manage to solve the problem?
For me it installed correctly, but it boot loops after installation. I've tried other types of Display (VGA, Vmware, SPICE) without succes.

It used to work with ProxMox 6 and 7 . I have worked successfully with Flex OS (with some screen issues) for more than 2 years. So I solved it eventually by backing up the virtual chromebooks, remove ProxMox 8 and reinstalling an older version of ProxMox: Version 7.1-7. Then restoring the virtual Chromebooks and don't touch the upgrade-button. Frustrating that upgrades are breaking working virtuals, but I guess Flex OS is considered a minor thing.