PVE 8.2.2 & 8.2.4 unexplained spontaneous reboots

[ohmantics]

I upgraded from 7.x to 8.2.2 and subsequently to 8.2.4. I am seeing unexplained spontaneous reboots.

The usual advice here is to assume I have bad RAM. The hardware in question is ECC, so I don't think that's the issue here, but rather some kind of kernel/driver/passthrough issue that appeared in 8.x since the upgrade is exactly when the issue appeared.

Assuming it's a software problem, would somebody please walk through the various ways to determine what's causing the reboots?

 

[alexnuno]

I'm curious to hear if you ever got to the bottom of this? I am in the same boat.

 

[ohmantics]

I'm curious to hear if you ever got to the bottom of this? I am in the same boat.

Nope, but it also hasn’t happened after I did a package update that got me to kernel 6.8.8. Could be coincidence.

EDIT: Spoke too soon. It just rebooted. Nothing in the logs.

 

[ohmantics]

Just happened again despite now being on 8.3.0/6.8.12-4.

 

[dakralex]

Hi!

What CPU is installed on the system? Is there some indicator that there was a sudden peak in temperature, voltage, power draw, etc.? If you can, a journal log from before and after the spontaneous boot could still be helpful to find an indicator for the issue.

 

[ohmantics]

Hi!

What CPU is installed on the system? Is there some indicator that there was a sudden peak in temperature, voltage, power draw, etc.? If you can, a journal log from before and after the spontaneous boot could still be helpful to find an indicator for the issue.

2x Xeon Gold 6136.

No physical indications logged that indicate anything special happened.

 

[alexnuno]

I actually found that disabling the pve-daily-update.timer has helped. I was having nodes restarting almost daily. Logs never really indicated what caused the reboots, but there was a fairly constant pattern of it happening after the pve-daily-update.service was initiated. I disabled it, figuring I can just run the updates manually when needed anyways and have found the cluster to be fairly stable, save for 1 recent reboot within the last week that I haven't looked into.
Reference logs:
Sep 05 02:40:10 aquila1 systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
Sep 05 02:40:16 aquila1 pveupdate[437895]: <root@pam> starting task UPID:aquila1:0006AEAC:005F46AC:66D97C80:aptupdate::root>
Sep 05 02:40:17 aquila1 iscsid[2396]: connect to 10.50.1.6:3260 failed (Connection refused)
Sep 05 02:40:20 aquila1 audit[4617]: AVC apparmor="ALLOWED" operation="open" class="file" namespace="root//lxc-501_<-var-li>
Sep 05 02:40:20 aquila1 kernel: audit: type=1400 audit(1725529220.219:100613): apparmor="ALLOWED" operation="open" class="f>
Sep 05 02:40:30 aquila1 audit[4617]: AVC apparmor="ALLOWED" operation="open" class="file" namespace="root//lxc-501_<-var-li>
Sep 05 02:40:30 aquila1 kernel: audit: type=1400 audit(1725529230.062:100614): apparmor="ALLOWED" operation="open" class="f>
Sep 05 02:40:31 aquila1 pvedaemon[351742]: worker exit
Sep 05 02:40:31 aquila1 pvedaemon[2644]: worker 351742 finished
Sep 05 02:40:31 aquila1 pvedaemon[2644]: starting 1 worker(s)
Sep 05 02:40:31 aquila1 pvedaemon[2644]: worker 438160 started
-- Boot 2d1442a9550545458b16ae9d280a74b6 --
Sep 05 09:19:55 aquila1 kernel: Linux version 6.8.12-1-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binu>
Sep 05 09:19:55 aquila1 kernel: Command line: BOOT_IMAGE=/ROOT/pve-1/@//boot/pve/vmlinuz ro ramdisk_size=16777216 root=ZFS=>
Sep 05 09:19:55 aquila1 kernel: KERNEL supported cpus:
Sep 05 09:19:55 aquila1 kernel: Intel GenuineIntel



For future references sake, should it be needed to be enabled again it would be just the following commands:

systemctl enable pve-daily-update.timer

systemctl start pve-daily-update.timer

systemctl enable pve-daily-update.service

If I should be concerned about any unintended consequences, please let me know.

 

[fabian]

that seems very unlikely to be honest, that service doesn't really do anything, and your reboot seems to be hours afterwards anyway?

 

[fiona]

Hi,
please make sure to use the latest kernel version and make sure you have latest BIOS updates and CPU microcode installed: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_firmware_cpu

You can also try the opt-in 6.11 kernel: https://forum.proxmox.com/threads/o...e-8-available-on-test-no-subscription.156818/