How do I shoot the watchdog? (constant reboots)

[stuartbh]

To whom it shall concern:

After years of running Proxmox, all of a sudden my nodes are rebooting every few hours. I believe it is due to watchdog timeouts and my network has not appreciably changed. Can anyone advise me as to how I can remediate this issue so my servers stop rebooting multiple times a day? I am sure there must be some setting I can adjust or test I can do to see why out of nowhere this has become an issue.

Thanks!

Stuart

 

[j.theisen]

Hi @stuartbh

thanks for posting on the forum and humorous choice of topic

The easiest way to stop the watchdog from rebooting your cluster would be to disarm HA.
Since PVE 9.2 there is an option for this under "Datacenter" -> "HA" -> "Disarm HA" -> "Ignore"
This should disable the HA services and in turn also the watchdog.

To troubleshoot the reboots themselves please provide the journal from before and after one of them so we may be able to see a possible cause:

journalctl --since="2026-MM-DD hh:mm" --until="2026-MM-DD hh:mm" | gzip > $(hostname)-journal.txt.gz

Please adjust the timestamps accordingly.

Best regards
Jonas

 

[stuartbh]

root@pve2-nuc7i7bnb:~# who -b
system boot 2026-08-11 06:33
root@pve2-nuc7i7bnb:~# journalctl --since="2026-08-11 06:23" --until="2026-08-11 06:33" | gzip > $(hostname)-journal.txt.gz
root@pve2-nuc7i7bnb:~#

 

[j.theisen]

Without having further details on the cluster this seems to be the problem:

Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] link: host: 4 link: 0 is down
Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 11 06:31:25 pve2-nuc7i7bnb corosync[1653]:   [KNET  ] host: host: 4 has no active links
Aug 11 06:31:26 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] Token has not been received in 3713 ms
Aug 11 06:31:27 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] A processor failed, forming new configuration: token timed out (4950ms), waiting 5940ms for consensus.
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Sync members[2]: 2 3
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Sync left[1]: 4
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] A new membership (2.22a92) was formed. Members left: 4
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [TOTEM ] Failed to receive the leave message. failed: 4
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: members: 2/1398, 3/5903
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: starting data syncronisation
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [QUORUM] Members[2]: 2 3
Aug 11 06:31:33 pve2-nuc7i7bnb corosync[1653]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [dcdb] notice: cpg_send_message retried 1 times
Aug 11 06:31:33 pve2-nuc7i7bnb pmxcfs[1398]: [status] notice: node lost quorum

Your node lost quorum since node 4 became unreachable.

How many nodes are in this cluster and what was node 4 doing today at 06:31?
What is the output of pvecm status?
Is the Corosync traffic on a separate link or are there other types of traffic flowing through that interface?

Since node 2 and 3 still were in a membership i think network issues directly on this servers are rather unlikely, but might still be the case.

Best regards
Jonas