Proxmox v6 started to freeze... Bare metal server has to be restarted to get all VMs running again...

[mstgeo]

Hi!

I started to have a problem with our proxmox v6 machine these days -- pls, see attached text file.

Is this the HARDWARE issue or rather a SOFTWARE one ? How can we fix and get rid that ? Anyone can help here ? Thanks in advance...

regards,
Grzegorz Leskiewicz

 

[Moayad]

Hi,

Proxmox VE 6.x is already EOL [0] we recommend upgrading to support version e.g., Proxmox VE 7.x or Proxmox VE 8.x. To upgrade to the Proxmox VE 7.x follow our wiki guide below link [1].

Regarding the `D state` you see in the syslog, do you see a high I/O in your Proxmox VE server before you see the `D state` message? Does the VM have enough memory?


[0] https://pve.proxmox.com/pve-docs/chapter-pve-faq.html
[1] https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

 

[mstgeo]

Hi!

During that time:
Iodelay: ~ 13, cpu usage: 52%.

It is a big machine with 1.5TB RAM and 96 CPUs + nvme raid-1 for system + 6 x ssd (raid-5) for VM storage +100 VMs mostly LXC and some KVM based, No vzdump during that time.

Any thoughts ?

regards,
G.

 

[aaron]

No other kernel messages that might indicate further where the issue is? I/O Errors or such?

Have you done a memory test?

 

[Moayad]

for VM storage +100 VMs mostly LXC

Do the LXCs run any kind of database?

 

[mstgeo]

Do the LXCs run any kind of database?

Hi. Yes, each one has mysql db running on it....
regards,
G.

 

[mstgeo]

I found this today:

Jan 10 21:13:19 proxmox-server kernel: [70971.086489] proc: Bad value for 'hidepid'
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jan 10 21:13:19 proxmox-server lxcfs[820788]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process

 

[Moayad]

Hi. Yes, each one has mysql db running on it....

Given the details of your current setup and the issues encountered, I recommend considering a migration from LXC to VMs. VMs typically offer better isolation and resource management, especially beneficial for database applications sensitive to I/O fluctuations. This change could potentially improve both performance and stability in your environment.

However, for more digging, please provide us with a full syslog that might help us to identify what the issue cause is. Additionally, as recommended above, consider upgrading your Proxmox VE node to the latest supported version.