[SOLVED] PVE unexpected reboots

[c3sro]

Hi,

a PVE node reboots unexpectedly.

PVE Facts:

• Kernel: Linux 6.5.13-5-pve
• Storage: 2xSSD (ZFS Raid 1)
• CPU: AMD Ryzen 9 7950X3D (16Core)
• RAM: 128GB

The crashes started after adding another VM that does some nested virtualization (VirtualBox inside PVE).

VMs on the PVE node:

• 6x VM with 4 CPUs each (Processor type: host)
• 1x VM with 8 CPUs doing nested virtualization (Processor type: host)

So all in all the PVE node is is overcommitted regarding CPUs (32 vCPUs, 16 Core CPU, 32 Threads CPU). All other resource consumption (RAM, Disk space) is low and not overcommitted.

Somewhere between 5 Minutes and 24 hours the PVE node unexpectedly reboots. There are no log entries in journalctl or /var/log/ regarding the crash (only boot of PVE node with filesystem checks).

Steps done to solve the problem:

• Enabled and tested kernel crash dumps: no crash dumps written
• Disabled reboot if kernel crash dump can't be written: still automatically reboots
• Provider ran a stress test on all components: no problems detected
• Added a custom CPU Type (x86-64-v4 with svm flag) (see https://forum.proxmox.com/threads/sudden-bulk-stop-of-all-vms.139500/#post-642039): still crashes / reboots
• Reduced the total number of vCPUs to 8 (see https://forum.proxmox.com/threads/sudden-bulk-stop-of-all-vms.139500/post-643308): PVE node doesn't crash anymore

Even though the workaround with reducing the total amount of vCPUs helped to stabilize the system this still isn't an acceptable solution for me as a lot of CPU resources are unused and the flexibility of the PVE node is quite limited.

The main problem I'm facing right now is that I can't tell what the root cause of the problem is. A kernel problem? A faulty CPU? A faulty PSU? ...

Has anyone had similar problems and solved them? Any further ideas to find the root cause?

 

[fiona]

Hi,
do you have the latest BIOS updates and CPU microcode installed? Do you have any special (CPU-related) settings enabled in BIOS? The opt-in 6.8 kernel might also be worth giving a shot.

 

[c3sro]

BIOS updates are managed by the provider, so thats nothing I can change (Pro WS 665-ACE, BIOS 1711 10/06/2023).

The server is running on microcode version 0x0a601206, which is the latest available version for this CPU.

As far as I know there are no special settings enabled in BIOS (primarily managed by provider again). Are there any specific settings I should check?

The 6.8 kernel might be worth a try. Do you know of any nested virtualization fixes in 6.6 - 6.8?

 

[fiona]

As far as I know there are no special settings enabled in BIOS (primarily managed by provider again). Are there any specific settings I should check?

I don't know what issue you have exactly, so unfortunately, I don't have any specific ones in mind.

The 6.8 kernel might be worth a try. Do you know of any nested virtualization fixes in 6.6 - 6.8?

Again, I don't know any specific ones.

 

[c3sro]

Update: The provider replaced the whole server (new CPU, memory, PSU, mainboard, ...). Unfortunately the server still crashes regularly.

As some people report similar problems (https://forum.proxmox.com/threads/sudden-bulk-stop-of-all-vms.139500/page-2) I think the reason isn't some faulty hardware (mainboard, memory, ...) or BIOS misconfiguration. A problem of the AMD Ryzen 9 7950X3D CPU seems much more likely.

It could be some CPU bug, Microcode bug or just a problem of the kernel. I've got not idea how to get to the root cause of the random crashes.

 

[JerryOH]

We also have this issue on 5 servers running Ryzen 9 7950x3d, each randomly lock up and the Proxmox login console is froze.

 

[Christoph Lechleitner]

We have had several reboots and lockups on Intel NUCs with i5 and i7 CPUs.

We'll upgrade to 8.2 (Kernel 6.8) in a few weeks and hope the problem will be gone ...

 

[c3sro]

Since upgrading from PVE 8.1 to 8.2 the server doesn't crash anymore.
Still have no clue what the root cause of the problem was.