My home server reboots itself between 8-9am every Monday

[Kevo]

I'm trying to figure out why my server reboots itself between 8am-9am every Monday.

I've tried looking through all the cron jobs and backups to see if I could find anything scheduled around that time and so far I've come up empty. The logs I've looked at don't turn up anything that doesn't appear to be a regularly occurring task so nothing stands out there.

I'm thinking my next step is going to be shutting down non-essential VMs for a couple of hours next Monday morning to try to find out if it's a particular VM causing the issue, but that seems unlikely.

I'm not really sure how to go about trying to log the issue. Is that possible, and is there any guide or info someone could point me to on debug logging of some sort.

Also, this start happening until some time after the server was up and running, but I'm not sure how long it was. It may have started after an update, but I think I'd have to reinstall to determine if a proxmox/debian update was the trigger.

Thanks

 

[killmasta93]

did you check the system logs? any kernel panics??

 

[Kevo]

Yes, I checked the system log(s) and looked for kernel dumps, but I couldn't find anything. Also started to setup kdump-tools, but I couldn't figure out after googling around for a bit how it's supposed to interact with systemd. I'm going to try hunting down more info on getting a kernel dump when I have some more time over the next couple of days. If you have any info on how to get it setup please point me in the right direction. TIA

 

[AlexLup]

Hi,
I have had this exact same issue since the latest maybe 3 kernels on all pve nodes!

Thing that did it for me was to update to the absolute latest kernel 4.15.18-7.

If you need to experiment with booting old kernels in PVE do this:
vi /etc/default/grub

Change
GRUB_DEFAULT=0
to
GRUB_DEFAULT="1>2"

and run update-grub

 

[Kevo]

I am on the latest kernel now. I also updated to the latest bios on the motherboard just in case something in the newer kernel was interacting oddly with the previous bios. This is on a Ryzen CPU so there's been some growing pains with those so maybe that will help as well. If not maybe I'll try going back to an older kernel from a couple of months ago and see if that makes a difference. I'd really like to figure out how to get a kernel dump when it crashes though if anyone has figured that out and can point me in the right direction.

 

[raku]

Do you use ZFS with swap on it? Try to disable swap.

 

[Kevo]

Well, I figured out what the issue was. I made sure to be near the server with terminals open to all my VMs and Proxmox watching top and logs. Thought the updates had fixed it once 9am rolled around, but about 6 minutes later everything stopped and I heard a beep, but it didn't sound like the tiny pc speaker beep from the server.

It was my UPS. Turns out the batteries are dead and the UPS's weekly self test was causing power to drop and make the server reboot.

I wonder if anyone knows of a bullet proof 1500VA sized UPS. I've had problems of various sorts with every single vendor I've tried. At least this UPS didn't shut everything off when the battery died. Unfortunately it also didn't raise any alarm about the battery dying either. I guess the voltage only sags under load and that's when it reads it as dead, but it doesn't trigger an alarm. I guess it thinks it just ran out of juice and needs to be recharged. Seems like there ought to be a UPS that can handle all the various ways batteries die, but so far I haven't found a brand that does so reliably. Seems they all can do weird things when the batteries fail.

Anyway, for now I have everything running off the surge only outlets while I wait for new batteries to arrive. Sorry for the wasted bandwidth.