VM freezes irregularly

[darkrayacc]

Linux 5.19.4-edge kernel solves the problem for me. No more VM crashes. It's running stable for the last 2 days.
What I have observed is that on other kernels. the ideal Memory usage was always more than 95% and with just a little bit more load, it used to crash. now the ideal memory usage is around 40-50%. It hits 95% in high load but settles down quickly. Earlier this was not happening.

 

[BarTouZ]

Despite a considerable evolution, I still have 1 VM (ubuntu 22.04) which freezes on the 3.

It is true that it takes more than 20 hours before they freeze, but we are not in 100%...

And you?

 

[diogo.f.m.7]

If I migrate to the 5.19.4-edge kernel, would it be possible to go back to the "default" kernel? in a future official pve?? kernel update.

 

[Holger Huo]

If I migrate to the 5.19.4-edge kernel, would it be possible to go back to the "default" kernel? in a future official pve?? kernel update.

Yes. You can simply remove edge kernel using apt or dpkg.

And you?

I've got the same problem. And some other forums have said that using an outdated kernel may increase stability so now I'm on 5.13.19-15 to test if it's ok.

 

[Holger Huo]

5.13.19-15

this did stablize openwrt but that was not the case for other vms... Seems that we have to get a perfect host-vm kernel combination(

 

[darkrayacc]

@BarTouZ has been working fine for the last 3 days. No VM Freeze.

 

[fabian]

thanks for the reports! if you are comfortable with running the pve-edge kernel until an official 5.19+ PVE kernel is available, that is likely a good-enough solution given that this issue only affects a very small, niche subset of hardware. if you want to find out more, doing a bisect of versions in-between is still an option, but it can only be done on affected hardware so the help I can offer there is minimal.

 

[gyrex]

thanks for the reports! if you are comfortable with running the pve-edge kernel until an official 5.19+ PVE kernel is available, that is likely a good-enough solution given that this issue only affects a very small, niche subset of hardware. if you want to find out more, doing a bisect of versions in-between is still an option, but it can only be done on affected hardware so the help I can offer there is minimal.

Thanks fabian. 5 days of uptime on all my VMs on the 5.19.x pve kernel here. I'd say whatever issue is present on earlier kernels that it's been fixed.

 

[Snk B]

thanks for the reports! if you are comfortable with running the pve-edge kernel until an official 5.19+ PVE kernel is available, that is likely a good-enough solution given that this issue only affects a very small, niche subset of hardware. if you want to find out more, doing a bisect of versions in-between is still an option, but it can only be done on affected hardware so the help I can offer there is minimal.

Do you have a window of time when this kernel will be available on the official branch?

 

[rRobbie]

Update from my NUC11ATKC4.

I updated to 5.19-edge kernel two days ago and the four testing VMs are running fine since then.

Many thanks to Fabian for releasing the edge kernel.

 

[fabian]

Do you have a window of time when this kernel will be available on the official branch?

no, nothing concrete yet.

 

[Holger Huo]

I have to say that this issue is not completely resolved on the 5.19 kernel..
Specifically, not with OpenWRT VM latest stable 22.03.0-rc6 (kernel 5.10.134). So this might still be a problem yet to be solved... @fabian

 

[BarTouZ]

For my part, it holds the road but with some hiccups all the same. But overall it's much better than before, that's for sure...

I have just restarted my pfsense VM on the n5105 to test, it works but I see that the Wireguard service is inactive and that I can connect well with my smartphone from 4G... Do you have that too?

 

[Holger Huo]

For my part, it holds the road but with some hiccups all the same. But overall it's much better than before, that's for sure...

I have just restarted my pfsense VM on the n5105 to test, it works but I see that the Wireguard service is inactive and that I can connect well with my smartphone from 4G... Do you have that too?

View attachment 40651

You mean that you can connect to wireguard via 4g but it shows not running? I assume that has something to do with pfSense instead of the kernel. Maybe you could check your configurations and have a look at wireguard's error log?

5.19 inarguably boosts stability and the issue I am having is only specific to OpenWRT, and considering the fact that OpenWRT is using quite an old kernel, maybe it has nothing to do with the hosts kernel at all, but OpenWRT's. However, as I'm using OpenWRT as my house's primary router, it was still unacceptable for 20-30s downtime anytime without warning. So I guess I still need to stick with 5.15 and wait for another release to see if it will help.

 

[Snk B]

Apparently this problem in VM's, only occurs with CPU's n5105?
Does the n6005 version have this same problem?

 

[Holger Huo]

Apparently this problem in VM's, only occurs with CPU's n5105?
Does the n6005 version have this same problem?

Yes. All 11-gen Celeron series are affected. The new pentium 7505 is said to be much more stable and performant( regarding single core performance and GPU, the drawbacks are that it is dual core(4 threads) and a little bit more expensive). I wish I had came across that nuc earlier...

(Anyhow, I had not that device so I can't guarantee it's stability. But this issue seems to be celeron-only)

 

[Snk B]

Yes. All 11-gen Celeron series are affected. The new pentium 7505 is said to be much more stable and performant( regarding single core performance and GPU, the drawbacks are that it is dual core(4 threads) and a little bit more expensive). I wish I had came across that nuc earlier...

(Anyhow, I had not that device so I can't guarantee it's stability. But this issue seems to be celeron-only)

Damn... Dual core for VM's is complicated, despite performing well.
I'm thinking of changing my n5105 minipc into an n6005, in order to "get out of" this problem without changing the main Proxmox kernel.
These VM crashes are unbearable.

 

[Holger Huo]

Damn... Dual core for VM's is complicated, despite performing well.
I'm thinking of changing my n5105 minipc into an n6005, in order to "get out of" this problem without changing the main Proxmox kernel.
These VM crashes are unbearable.

If you don't need ultra low power maybe some i3/i5 devices are also suitable.

This kernel bug wasted me many hours and now ive decided to settle down and use it solely for routing and simple network management maybe sometime in the future, there could be a fix for this device, but now I will just give up on it.

 

[Holger Huo]

If you don't need ultra low power maybe some i3/i5 devices are also suitable.

This kernel bug wasted me many hours and now ive decided to settle down and use it solely for routing and simple network management maybe sometime in the future, there could be a fix for this device, but now I will just give up on it.

And the thing that I've really learned from it is I will definitely carry-out full-fledge research on the device I'm going to buy

 

[gyrex]

Damn... Dual core for VM's is complicated, despite performing well.
I'm thinking of changing my n5105 minipc into an n6005, in order to "get out of" this problem without changing the main Proxmox kernel.
These VM crashes are unbearable.

It's pretty easy to upgrade the kernel and once you do it, you can expect rock solid stability.