Recent content by coenvl

The problem that this thread refers to actually has been solved. It appears a bug was found and fixed in the kernel relating to the virtualization process. The fix to this bug in particular is in: 6.2.16-11~bpo11+2 of the 6.2 opt-in kernel on PVE 7/Bullseye 6.2.16-12 of the 6.2 (current)...

Sorry it took me a while, but I am happy to confirm that for me switching to kernel version 6.2.16-11-bpo11-pve, indeed seems to have resolved the problem. I was able to reproduce the bug before, but not after applying the upgrade. The mmu_invalidate_seq counter is now well into the 3 billion...

I can confirm the bug on my cluster as well on my cluster with kernel 6.2.11-1, not sure exactly how long it took to trigger it, but it was less than 7 hours. That is, for the Debian VMs, when I keep switching ballooning between 4GB and 100GB of RAM.

I am running the test to see if I can reproduce the freeze as well. It will take a while I think, since running it for 2 hours only got the mmu_invalidate_seq value to ~350M. I also found that for some reason I have to repeat the "balloon $amount" command within every loop iteration, otherwise...

Wow, that would be nice. Does this mean that we can expect an update coming soon that incorporates this bugfix, or better yet, is it already available?

It's been a while since I posted an update, which is good news. We haven't had any random freezes since half of July, so it seems we are stable again. I cannot really tell what solved the issue but I can say that I have now: Disabled ballooning on VMs that used to crash, but still using it on...

Thanks jens-maus for your findings. I will definitely try setting `mitigations=off` the next time I have another hanging VM. I think I have to reboot the host, so it involves migrating all other running VMs first. I hope it solves my problem as well, since we have never seen the issue on any of...

I am running kernel version 6.2 (see for example my post from June 20th). I never experienced this problem with kernel 5.15. in fact, the problem started occurring when I started using the opt-in kernel version 5.19, and later when the opt-in version changed to 6.2. The reason I started using...

Yes it seems that more people are facing similar issues, in various (slightly differently named) threads. I can also confirm that on older versions of proxmox I have never seend this happening before, running for about 4 years. Perhaps anybody already switched to version 8, and can share their...

How do you define "from time to time"? I think this might be the same issue as here: https://forum.proxmox.com/threads/vms-freeze-with-100-cpu.127459. It has not been resolved (for me at least), but perhaps there are some pointers which may help you. Or at the least it has some instructions to...

So this morning I caught another one. Here is the output of the logs, but I don't think it looks anything different from the last time. strace: strace -c -p $(cat /var/run/qemu-server/115.pid) strace: Process 195052 attached ^Cstrace: Process 195052 detached % time seconds usecs/call...

Thank you so much already for helping out here. The host that the log was from this morning has 2 Intel Xeon Silver 4114T CPUs, that are using the Skylake architecture. The problem also occurs on other nodes that have Intel Xeon CPU E5-2640 v4 CPUs with a Broadwell architecture. @fweber, as...

I also got another frozen VM today. Output of the commands you asked for are as follows: Strace: strace -c -p $(cat /var/run/qemu-server/144.pid) strace: Process 3703171 attached ^Cstrace: Process 3703171 detached % time seconds usecs/call calls errors syscall ------ -----------...

So I have a similar experience. I recently posted on the forum as well (see here), but think that this thread is about the exact same issue. I started seeing this pattern after upgrading from proxmox 6 to 7, and have freezing VMs roughly once a week . I already tried some fixes on the host...

Dear all, I have a cluster of 3 proxmox nodes, currently 45 VMs in a research setting. I have been operating this cluster for about 6 years now, started out with proxmox 4, and have been upgrading every time to the latest release according to the upgrade protocol without too much trouble...