Proxmox VPS stuttering/frezing

R

radomirpolach

Member
Nov 5, 2021
12
0
6
37
I have several Proxmox servers and on 2 separate servers running Proxmox 8.3.x, I have serious issues with my VMs temporarily stuttering/freezing. It was first notice by high pings repeating in 60-90 seconds patterns. But after running echo date in loop, I am pretty certain that this not a network issue, but whole VM is temporarily stuttering/freezing. When freeze occurs echo date in loop just doesn't even print anything. There is nothing dmesg in VM or in Proxmox host. There is nothing else running on those servers and VMs are running fully updates Ubuntu 24.04 LTS. If I restart VM, everything is fine and takes like a day or so for the problem to reappear.

This is the behavior measured using ping, you can see restart in the night, then almost whole day goes by without any issue and then it is slowly becoming worse:

1744194144633.png

Restarting VM again solves it. The issue is progressively getting worse the longer it runs. I don't see the issue anywhere on Proxmox 7.4. The VM are pretty big 70-80 threads, 700GB RAM, both servers running SQL databases as well apache/php/haproxy and other containers. I tried changing disk controller to single and vice versa, disabling and enabling iothread, disabling and enabling offloading in network driver. No change.

What the hell is going on? I naver seen anything like this. I would like to move my Promox 7.4 to 8.x, but if 8.x is this buggy, I am not sure what to do.

Any ideas what could be the issue?
 
Last edited:
I wrote a script that does timestamp in a loop on the VM:

Code: 
Previous: 1744891845
Current : 1744891849
Diff    : 4 seconds

There is no sleep nothing between commands and there are basically 4+ seconds lags in loop. So VM basically freezes.
 
Last edited:
_gabriel said:
VPS isn't the correct word.
VPS are VM rent by providers.
You should replace all words VPS by node (or at least host).

CPU over allocation ?

Edit :
Seems related ?
Post in thread 'Proxmox 8.0 / Kernel 6.2.x 100%CPU issue with Windows Server 2019 VMs' https://forum.proxmox.com/threads/p...th-windows-server-2019-vms.130727/post-663789
Click to expand...
Load is very low 5-10 (for 68 threads in VM). So I doubt it can be related to any CPU provisioning, high load etc. Also no CPU soft locks.
Wrong terminology isn't really the issue here.
 
Last edited:
Many are allergic to real VPS...

Have you read the post ?
There was issue about numa and KSM.
What about you KSM usage ?
What about your CPU allocation ?
vCPUs total + vCPU per VM
edit: What are physical CPU model ?
 
Last edited:
_gabriel said:
Many are allergic to real VPS...

Have you read the post ?
There was issue about numa and KSM.
What about you KSM usage ?
What about your CPU allocation ?
vCPUs total + vCPU per VM
Click to expand...
80 thread, 76 allocated to VM, 2 to mini test VM, 2 left to Proxmox.

I tried just now. Disabling numa balancing didn't help.
But disabling ksm and unmerging pages may have solved it. Maybe too soon to say. But it looks promising.
 
Last edited:
_gabriel said:
if CPU are old, mitigations=off helps too.
at host level
or
at VM level, selecting vCPU type "x86-64-v2-aes"
Click to expand...
I was testing that as well, seems to do something, but not solve the issue, this would be 100% ksm, it seems.

Thank you very much.
 
Last edited:
You must log in or register to reply here.
Top Bottom