How do I shoot the watchdog? (constant reboots)

S

stuartbh

Well-Known Member
Dec 2, 2019
143
19
58
61
To whom it shall concern:

After years of running Proxmox, all of a sudden my nodes are rebooting every few hours. I believe it is due to watchdog timeouts and my network has not appreciably changed. Can anyone advise me as to how I can remediate this issue so my servers stop rebooting multiple times a day? I am sure there must be some setting I can adjust or test I can do to see why out of nowhere this has become an issue.

Thanks!

Stuart
 
Hi @stuartbh

thanks for posting on the forum and humorous choice of topic ;)

The easiest way to stop the watchdog from rebooting your cluster would be to disarm HA.
Since PVE 9.2 there is an option for this under "Datacenter" -> "HA" -> "Disarm HA" -> "Ignore"
This should disable the HA services and in turn also the watchdog.

To troubleshoot the reboots themselves please provide the journal from before and after one of them so we may be able to see a possible cause:
Code: 
journalctl --since="2026-MM-DD hh:mm" --until="2026-MM-DD hh:mm" | gzip > $(hostname)-journal.txt.gz
Please adjust the timestamps accordingly.

Best regards
Jonas
 
root@pve2-nuc7i7bnb:~# who -b
system boot 2026-08-11 06:33
root@pve2-nuc7i7bnb:~# journalctl --since="2026-08-11 06:23" --until="2026-08-11 06:33" | gzip > $(hostname)-journal.txt.gz
root@pve2-nuc7i7bnb:~#
 

Attachments

Without having further details on the cluster this seems to be the problem:
Code: 
Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] link: host: 4 link: 0 is down
Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] host: host: 4 has no active links
Aug 11 06:31:26 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] Token has not been received in 3713 ms
Aug 11 06:31:27 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] A processor failed, forming new configuration: token timed out (4950ms), waiting 5940ms for consensus.
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Sync members[2]: 2 3
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Sync left[1]: 4
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] A new membership (2.22a92) was formed. Members left: 4
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] Failed to receive the leave message. failed: 4
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: members: 2/1398, 3/5903
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: starting data syncronisation
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Members[2]: 2 3
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: cpg_send_message retried 1 times
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [status] notice: node lost quorum

Your node lost quorum since node 4 became unreachable.

How many nodes are in this cluster and what was node 4 doing today at 06:31?
What is the output of pvecm status?
Is the Corosync traffic on a separate link or are there other types of traffic flowing through that interface?

Since node 2 and 3 still were in a membership i think network issues directly on this servers are rather unlikely, but might still be the case.

Best regards
Jonas
 
  • Like
Reactions: Johannes S and Kingneutron
Jonas, et alia:

I have cleaned up my cluster, removing node 1 and fixing node 4 and it seems now to be stable.

I am getting ready to start taking the smaller NUC cluster members I have out of service and migrate to using more x3650 server grade hardware I have. One reason is that the x3650 servers can have 10GB Ethernet cards added and I plan to use the 10GB network cards to dedicate to running ceph on in the future.

Stuart
 
SteveITS said:
Ceph volume can impact latency and corosync is very sensitive, hence the recommendation to use a dedicated NIC. Note Corosync can also use additional/backup links.
Click to expand...
Steve,

At the moment, I am not using CEPH and it was not the impetus as to why the watchdog was barking. Currently I use an NFS mount from my TrueNAS server. Eventually, I will implement CEPH.

When my TrueNAS server needs to be rebooted, I shut down all my VM’s prior. However, my pre-CEPH interim solution is going to be to set up a second TrueNAS server so I have a secondary NFS mount point that I can migrate the qcows to live and reboot the other TrueNAS server.

Stuart
 
SteveITS said:
Ceph volume can impact latency and corosync is very sensitive, hence the recommendation to use a dedicated NIC. Note Corosync can also use additional/backup links.
Click to expand...

How can corosync be setup to use backup links?
Is there also a way to configure the time on the watchdog?
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back