Watchdog Reboots

X

XN-Matt

Renowned Member
Aug 21, 2017
95
7
73
44
Something we've seen in version 9 is an increase in watchdog reboots - in fact from none to many.

Last few entries of the journal show:

```
Jan 22 04:39:09 hv-5-i watchdog-mux[1504]: client watchdog is about to expire
Jan 22 04:39:09 hv-5-i systemd-journald[841]: Received client request to sync journal.
Jan 22 04:38:07 hv-5-i pveupdate[2200016]: <root@pam> starting task UPID:hv-5-i:00219207:010ADDDA:6971A9AF:aptupdate::root@pam:
Jan 22 04:38:08 hv-5-i pveupdate[2200071]: update new package list: /var/lib/pve-manager/pkgupdates
Jan 22 04:38:13 hv-5-i pveupdate[2200016]: <root@pam> end task UPID:hv-5-i:00219207:010ADDDA:6971A9AF:aptupdate::root@pam: OK
```

The node is not losing network connection as far as we know but i'm at a loss to determine if we can isolate why.

Even changed from softdog to iTCO_wdt to see if this helps but it rebooted again today.

It had one active VM at the time, deliberately to see if it would happen with minimal load.

Setup is 4 x ceph nodes and 3 x compute nodes running the VMs. This is one of the latter. Each server has primary/secondary links for WAN, ceph and pve nets. Switch shows no port drops or losses either.

I see this has come up a few times and disabling isn't workable as we use HA in a cluster.. but also this had been ultra stable for over a year before the upgrade to 9.x
 
Last edited:
No entries around the time of the reboot.

Code: 
Jan 20 04:04:06 hv-5-i corosync[1826]:   [QUORUM] Sync members[7]: 1 2 3 4 5 6 7
Jan 20 04:04:06 hv-5-i corosync[1826]:   [QUORUM] Sync joined[6]: 1 2 3 4 6 7
Jan 20 04:04:06 hv-5-i corosync[1826]:   [TOTEM ] A new membership (1.759) was formed. Members joined: 1 2 3 4 6 7
Jan 20 04:04:06 hv-5-i corosync[1826]:   [QUORUM] This node is within the primary component and will provide service.
Jan 20 04:04:06 hv-5-i corosync[1826]:   [QUORUM] Members[7]: 1 2 3 4 5 6 7
Jan 20 04:04:06 hv-5-i corosync[1826]:   [MAIN  ] Completed service synchronization, ready to provide service.
-- Boot 6fd6ce7cb48041ec96100b83e762c2cc --
Jan 22 04:42:05 hv-5-i systemd[1]: Starting corosync.service - Corosync Cluster Engine...
Jan 22 04:42:05 hv-5-i (corosync)[1812]: corosync.service: Referenced but unset environment variable evaluates to an empty string: COROSYNC_OPTIONS
Jan 22 04:42:05 hv-5-i corosync[1812]:   [MAIN  ] Corosync Cluster Engine  starting up
Jan 22 04:42:05 hv-5-i corosync[1812]:   [MAIN  ] Corosync built-in features: dbus monitoring watchdog augeas systemd xmlconf vqsim nozzle snmp pie relro bindnow
Jan 22 04:42:05 hv-5-i corosync[1812]:   [MAIN  ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Jan 22 04:42:05 hv-5-i corosync[1812]:   [MAIN  ] Please migrate config file to nodelist.

Server rebooted around 04:39:09 on 22nd Jan.

Networks/NICs as described but this appears to not be the issue.
 
We are experiencing the same issue as described by XN-Matt while upgrading nodes in a cluster one-by-one from 8.4 to 9.1. Just after rebooting one of the nodes as part of completing the dist-upgrade almost all other nodes (mix of v8 and v9) in the cluster fenced themselves:

Code: 
2026-01-22T15:10:47+0100 bt-01-node-b-wp-a watchdog-mux[2119]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:47+0100 bt-01-node-b-wp-a watchdog-mux[2119]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:47+0100 bt-01-node-c-wp-a watchdog-mux[2149]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:37+0100 tp-01-node-c-wp-a watchdog-mux[1705]: client watchdog is about to expire
2026-01-22T15:10:47+0100 tp-01-node-c-wp-a watchdog-mux[1705]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:38+0100 tp-01-node-d-wp-a watchdog-mux[1724]: client watchdog is about to expire
2026-01-22T15:10:48+0100 tp-01-node-d-wp-a watchdog-mux[1724]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:37+0100 tp-02-node-a-wp-a watchdog-mux[1761]: client watchdog is about to expire
2026-01-22T15:10:47+0100 tp-02-node-a-wp-a watchdog-mux[1761]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:41+0100 tp-02-node-b-wp-a watchdog-mux[1748]: client watchdog is about to expire
2026-01-22T15:10:51+0100 tp-02-node-b-wp-a watchdog-mux[1748]: client watchdog expired - disable watchdog updates
2026-01-22T15:10:37+0100 tp-02-node-c-wp-a watchdog-mux[1726]: client watchdog is about to expire
2026-01-22T15:10:47+0100 tp-02-node-c-wp-a watchdog-mux[1726]: client watchdog expired - disable watchdog updates

It's also strange that on some nodes `watchdog-mux` didn't even log that it was about to expire.
 
Now you mention it, we saw that on our last upgrade.

Nodes drying during apt-get dist-upgrade. Never seen that before. We just thought it was because they were busy.
 
You must log in or register to reply here.
Top Bottom
Back