VM freezes irregularly

[gyrex]

I'm running my OPNSense VM with 3 bridged NICs (1 LAN and 2 WANs, no PCI passthrough). CPU Type is HOST, Bios is OVMF (UEFI) and Machine Type Q35. Are you experiencing VM freezes with a similar setup than the above? I could try creating other OPNSense VMs with different specs if you think is worth it.

It doesn't matter what the VM settings are (I've tried just about every permutation), freezes are inevitable, less frequent on freebsd but still inevitable.

 

[BarTouZ]

I have the same experience as gyrex. moreover,


I have the PfSense vm which has the 4 nics and everything and I noticed that the wireguard service being considered down but I knew it still connects from my phone to my PfSense vm via wireguard...

bug ?

 

[wolan]

Crashes happen on every kernel from 5.10.0 up until 5.17.0. So does it still seem to be caused by a kernel bug?
I have a semi-reliable way to crash a VM, about half of soft reboots of my Home Assistant VM cause a freeze just after screen goes blank after "system going to halt" message. All soft shutdowns complete properly though. Does that ring any bell?

 

[gyrex]

Crashes happen on every kernel from 5.10.0 up until 5.17.0. So does it still seem to be caused by a kernel bug?
I have a semi-reliable way to crash a VM, about half of soft reboots of my Home Assistant VM cause a freeze just after screen goes blank after "system going to halt" message. All soft shutdowns complete properly though. Does that ring any bell?

Have you tried, or could you try 5.19.3?

 

[BarTouZ]

I come with bad news for me...
After 3 days of running I have a VM that froze on ESXi as well.

After a reset of the VM, here we go again.
Looks like we have a similar phenomenon finally, maybe ESXi is more flexible but personally, it makes me freeze a VM all the same...

And of course, it has to happen the night Madame comes home what a bad luck

 

[wolan]

Have you tried, or could you try 5.19.3?

Tried it this morning, no change.

 

[gyrex]

I come with bad news for me...
After 3 days of running I have a VM that froze on ESXi as well.

After a reset of the VM, here we go again.
Looks like we have a similar phenomenon finally, maybe ESXi is more flexible but personally, it makes me freeze a VM all the same...

And of course, it has to happen the night Madame comes home what a bad luck

That's really disappointing to hear. I haven't had any freezes on my VMs on ESXi as of yet.

 

[BarTouZ]

Hello,
I had again a freeze on the same VM around 1:00 am
I look on ESXi to see if there are already people who had this problem and obviously, yes, I am not the only one.
It would come from the NVME module: https://kb.vmware.com/s/article/88025

So, does your N5105 also work with an NVME? For me, it is.

I don't know if that could be a lead.

 

[gyrex]

Hello,
I had again a freeze on the same VM around 1:00 am
I look on ESXi to see if there are already people who had this problem and obviously, yes, I am not the only one.
It would come from the NVME module: https://kb.vmware.com/s/article/88025

So, does your N5105 also work with an NVME? For me, it is.

I don't know if that could be a lead.

I had issues with nvme drives on vmware so I replaced it with an SSD SATA disk.

 

[BarTouZ]

OK,
So since you switched from NVME to SATA, you haven't had any problems with VMWare, interesting...

I assume you did the same test with Proxmox and the freezes continued?

 

[gyrex]

OK,
So since you switched from NVME to SATA, you haven't had any problems with VMWare, interesting...

I assume you did the same test with Proxmox and the freezes continued?

No freezes on ESXi since changing to SSD SATA. Didn't try on Proxmox because others on here mentioned they were running a variety of storage hardware and SANs and had the same issue.

 

[wolan]

I am also running on NVME drive and no SATA on my box

 

[BarTouZ]

I wonder if the problem would not come from the NVME, in any case from the support of the NVME in the end...

I had 1 freeze after 3 days on ESXi and another 1 after not even 24 hours and I'm running on NVME...

Gyrex runs on ESXi but on SATA and it has nothing...

You also have freezes and you are on NVME...

The last test would be to switch to proxmox and SATA.

For my part, I moved 1 VM to external storage on ESXi, which runs on USB3, if I no longer have a freeze on this VM, it could actually come from the NVME for once...

What do you think ?

 

[wolan]

I will try a USB storage for the VMs this evening, but I doubt that this is the solution. The NVME and it's modules are still present in the system, and I don't think the VMs would access it directly, but I may be wrong

 

[BarTouZ]

Yes you are right, what is weird is that both ESXi and Proxmox work perfectly on the host...
It's really virtualization that crashes / freezes from time to time and personally, I have 4 vm on the host and they don't crash all 4 at the same time either...
So on reflection, if it was really a disk access error (nvme) or something else, it would happen on all VMs, but it is not...

We are going in circles

 

[gyrex]

Yes you are right, what is weird is that both ESXi and Proxmox work perfectly on the host...
It's really virtualization that crashes / freezes from time to time and personally, I have 4 vm on the host and they don't crash all 4 at the same time either...
So on reflection, if it was really a disk access error (nvme) or something else, it would happen on all VMs, but it is not...

We are going in circles

If you read this and other threads, you'll find others who've run different storage and run into the same issue. There's people running different processor types and NVMe drives without issue.

I still think it's some kind of a KVM/qemu issue in the host kernel.

 

[wolan]

I am running kernel 5.19.3 for ~30 hours, the only crash I got was during the VM reboot, no freeze during normal usage.
Maybe the reboot freeze is a separate issue? Got some logs on VM monitor screen, that I have previously only seen when the VM froze (screenshot attached).
Attached also a log file from when the machine froze at reboot, it's just a sample, the log is flooded with these messages at reboot freeze.
Does that give you any idea?

 

[gyrex]

I am running kernel 5.19.3 for ~30 hours, the only crash I got was during the VM reboot, no freeze during normal usage.
Maybe the reboot freeze is a separate issue? Got some logs on VM monitor screen, that I have previously only seen when the VM froze (screenshot attached).
Attached also a log file from when the machine froze at reboot, it's just a sample, the log is flooded with these messages at reboot freeze.
Does that give you any idea?

This is promising news. I thought you said that this kernel wasn't any good in this post? : https://forum.proxmox.com/threads/vm-freezes-irregularly.111494/post-492979

 

[Snk B]

Here on my host, pfsense is solid so far.
But, my Ubuntu Server and Windows Server 2022 machines keep having kernel panic
I had a crash of the ubuntu kernel at 07:30 in the morning and the windows crash at 07:00 in the morning.
I'm already thinking about testing Windows Server 2022 bare metal with Hyper-V and uploading these machines both pfsense and ubuntu server into it to see if it has more stability.

 

[wolan]

This is promising news. I thought you said that this kernel wasn't any good in this post? : https://forum.proxmox.com/threads/vm-freezes-irregularly.111494/post-492979

My VM is freezing upon soft reboot on every kernel, 5.19.3 included. I thought it might be the same issue as the freeze during normal work, but maybe it's a separate one.