Random VM crashes with a SPICE vm: QEMU free(): corrupted unsorted chunks

Worth noting, the VM of mine that keeps crashing is a debian 11 VM, not windows. So guess would be not guest related.
 
I don't have storage latency and there are still random VMs crashes with messages in logs like yours. I only use SPICE on Windows Server VMs of which I don't have many, but sudden stops of machines are irritating. This started happening after the last updates.
same. The are no latency problem today, but two VMs crashes
 
Hrmm... it looks like my coredumps timeout while being taken, for some wacky reason... The coredump program gets sigkilled...?
 
Unfortunately, all the provided backtraces show crashes in different locations in the code. So my best guess is that it is a heap corruption, which is notoriously difficult to debug. What you might still try is to gather debug messages by doing
Code:
export G_MESSAGES_DEBUG=all
qm start <ID>
There will be some initial messages in the terminal, the rest should be logged to the system journal.

You might also want to report the issue to Debian or to the SPICE project upstream, maybe they have an idea for how to debug this further: https://gitlab.freedesktop.org/groups/spice/-/issues
 
what I noticed in the logs after the crash of the virtual machine (debug mode is enabled, as mentioned above)
 

Attachments

  • logs_20240423111352.txt
    23.3 KB · Views: 3
Good afternoon!
Just last week, a similar problem appeared, I notice the same errors in the logs as the people above
 
I have less problems on my slower machines. When I ran my VM's on an X5670, I had zero crashes ever. When I moved them to a 6140, sometimes it would crash 4-5 times a day. Currently I have the VM sitting on an e5-2690v2 and it's been happy for a week now. Previously it did crash once each on an e5-2670v0 and an e5-2670v2.

Heavy graphics seems to be what triggers it. I often trigger it in one of three ways:

1) I preview mkv files in thunar. On the 6140 I would usually crash it in about an hour of fiddling like this.
2) Firefox. Switching tabs, scrolling on a web page, loading new web page. The web page doesn't need to be graphically intense. I've crashed it while scrolling down in netbox just filling out a form.
3) pulling up a VNC from another box and fiddling around in there.

I've never had the vm crash overnight, or when idle, even with firefox running. I've never triggered it by just moving windows around or fiddling about in emacs. It's always heavy graphics use. I haven't tried something like running a demo overnight on it or something. I have 4 nodes of 6140's, it crashes the same on all 4 nodes. No other non-spice VM's crash, on any node, regardless of workload.

The VM that crashes does NOT use virgl. I have a few VM's with virgl, but they are very lightly used, so I cannot say one way or another if they are crash prone.

Worth noting, I believe one of the other replies to this post was experiencing this on a windows VM, whereas mine is debian, so it looks like OS is not a factor.
 
Hi,
what I noticed in the logs after the crash of the virtual machine (debug mode is enabled, as mentioned above)
these are OOM (out-of-memory) debug messages. How much memory do you have assigned in the display settings? You could try setting more and see if the crashes get rarer. That would be a hint that it's really related to the OOM-Handling for SPICE.

I'm able to get lots and lots of OOM-Messages from SPICE when configuring the minimum amount of display memory the UI allows, i.e. 4MiB. But was still not able to reproduce a crash even with videos running and tab-switching in Firefox.
 
With a memory capacity of more than 16 MB, the picture simply does not display.
I'ill try to set 32 MiB to all users and keep an eye on system crashes.
My current settings are as follows:
1713883193905.png
 
1713884023910.png

My setup for the VM that keeps crashing (also, the only one I heavily use)

Curious, is everyone having this problem on multi-monitor? All my other spice VM's are single monitor.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!