Proxmox freeze on Intel NUC

[jdruwe]

Helly everyone, I am experiencing freezes (2 in the last 2 days) that require me to reboot the Intel NUC (https://www.intel.com/content/www/us/en/products/boards-kits/nuc/kits/nuc8i3beh.html) manually. I can see under summary of my node that it happened this time at 20:20 in the evening. The Syslog shows logs starting from after the manual reboot. I have no idea where to start, can I enable logging of some sort? I noticed that there is a new bios version for the nuc, is it worth trying?

 

[n1nj4888]

some questions / pointers below.

(1) Was this a one-off or happening multiple times?
(2) Anything relevant in the Syslog before the freeze?
(3) What was on the screen when the NUC froze? Did you manage to capture a screenshot/photo?
(4) Have you tested the RAM in the NUC using memtest?
(5) I’d update the BIOS to the latest version
(6) i was experiencing some kernel panics / freezes on my NUC8i5beh last year but recent bios / Proxmox / kernel updates have made the NUC very stable. Given my NUC was headless, I wasn’t able to see what was on the screen at the point of kernel panic so I wrote a guide at the following thread to capture the kernel panic logs on a remote server using netconsole:

https://forum.proxmox.com/threads/proxmox-host-random-freeze.54721/post-252772

 

[jdruwe]

Thank you so much for your detailed response, here are my answers:

(1) Was this a one-off or happening multiple times?
It happened 1 time before

(2) Anything relevant in the Syslog before the freeze?
No not really, I did see a lot of:

Started Proxmox VE replication runner.

So I masked that service as I have no replication set up at the moment.

(3) What was on the screen when the NUC froze? Did you manage to capture a screenshot/photo?
I did not manage There is no screen connected it's headless, I just connect to the admin interface on my macbook.

(4) Have you tested the RAM in the NUC using memtest?


It's still running atm but this is the result after 2 passes, seems fine.

(5) I’d update the BIOS to the latest version
I just updated to the latest version and i haven't had any crash since (been running now for straight 10 hours I think)

(6) i was experiencing some kernel panics / freezes on my NUC8i5beh last year but recent bios / Proxmox / kernel updates have made the NUC very stable. Given my NUC was headless, I wasn’t able to see what was on the screen at the point of kernel panic so I wrote a guide at the following thread to capture the kernel panic logs on a remote server using netconsole:
https://forum.proxmox.com/threads/proxmox-host-random-freeze.54721/post-252772

I would like to that if the error still occurs but I don't have server laying around, would a macbook or a windows machine also work?

Is there a difference between Proxmox and kernel updates? Or by kernel do you mean those on the VM's?

 

[Dark26]

try to disable c6 state in the bios

 

[n1nj4888]

I would like to that if the error still occurs but I don't have server laying around, would a macbook or a windows machine also work?

Is there a difference between Proxmox and kernel updates? Or by kernel do you mean those on the VM's?

Hi! You could run a VM on the windows machine using VirtualBox and use that VM as the netconsole target but obviously the VM needs to be running and awaiting a kernel panic on the Proxmox node.

In terms of differences between Proxmox and kernel updates , the kernel updates are just a subset of the Proxmox updates but yes I meant that clicking “Update” in the web ui and updating Proxmox generally may have also solved my previous issue

 

[jdruwe]

Hi! You could run a VM on the windows machine using VirtualBox and use that VM as the netconsole target but obviously the VM needs to be running and awaiting a kernel panic on the Proxmox node.

In terms of differences between Proxmox and kernel updates , the kernel updates are just a subset of the Proxmox updates but yes I meant that clicking “Update” in the web ui and updating Proxmox generally may have also solved my previous issue

Thanks for the information, I'll run a VM on my windows machine if I experience such a crash again even after all the updates I did (bios and now th proxmox provided updates via the UI). Did you disable the c6 state when you were having issues as @Dark26 mentioned or did you keep it as it was and the issue resolved without touching it?

 

[n1nj4888]

I can’t recall whether I changed anything with the cstate in the BIOS - I know I looked into it at the time but I don’t recall amending anything in the BIOS specifically for this.

If you can point me to the cstate settings in the BIOS, I can tell you what mine are set to...

 

[jdruwe]

I can’t recall whether I changed anything with the cstate in the BIOS - I know I looked into it at the time but I don’t recall amending anything in the BIOS specifically for this.

If you can point me to the cstate settings in the BIOS, I can tell you what mine are set to...

I am unable to find the setting, I did find this response from an Intel employee:

"There is no such an option in BIOS that explicitly says: C states or P States -disable or enable-. The only options available that could help are performance and power related tools but they won't necessarily disable any of these states.

I would recommend that you check at the operating system level. There are a couple of commands available for Linux* operating systems and for Windows* it usually depends on how "Power Options" à "Advanced Settings" à "Minimum and Maximum processor state" settings are configured.

This is also third party tools that will show you disabled/enable C and P States. This could be useful."