[SOLVED] Watchdog rebooted server at random moment - how to debug?

[alexc]

I use Supermicro server with PVE 6.2. I did the watchdog setup this way:

/etc/default/pve-ha-manager:
WATCHDOG_MODULE=ipmi_watchdog

/etc/modprobe.d/ipmi_watchdog.conf:
options ipmi_watchdog action=power_cycle panic_wdt_timeout=10

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet nmi_watchdog=0"

here is what I see in output:

# ipmitool mc watchdog get

Watchdog Timer Use:     SMS/OS (0x44)
Watchdog Timer Is:      Started/Running
Watchdog Timer Actions: Power Cycle (0x03)
Pre-timeout interval:   0 seconds
Timer Expiration Flags: 0x00
Initial Countdown:      10 sec
Present Countdown:      9 sec

Server run for months perfectly, and today it rebooted at random moment, with no reasons for.

If there any way to debug what was the core reason for?

 

[mira]

Check the syslog (/var/log/syslog[.X.gz]). Are there any corosync and watchdog messages?

 

[alexc]

No corosync, and watchdog mentioned only as reboot (boot) was in progress so grep returns only these:

Dec 15 11:49:07 hostname kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.34-1-pve root=/dev/mapper/pve-root ro quiet nmi_watchdog=0
Dec 15 11:49:07 hostname kernel: [    0.594703] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.34-1-pve root=/dev/mapper/pve-root ro quiet nmi_watchdog=0
Dec 15 11:49:07 hostname systemd[1]: Started Proxmox VE watchdog multiplexer.
Dec 15 11:49:07 hostname watchdog-mux[1479]: Loading watchdog module 'ipmi_watchdog'
Dec 15 11:49:07 hostname watchdog-mux[1479]: Watchdog driver 'IPMI', version 1

No traces of watchdog actions / warnings / host hangs.

 

[mira]

Please provide the complete syslog from ~15 minutes before and after the reboot.

 

[alexc]

So to say, watchdog was right: RAID card failed and IO become too slow and buggy and stucky, so IPMI based watchdog detected is as hangs and rebooted the server.

Thank you anyway!

 

[aaron]

Thanks for sharing what the problem was!