PVE 4 with HA

[dietmar]

There is an easy way to test if the watchdog works correctly:

# echo 1 >/dev/watchdog

This should trigger a reboot within 60 seconds. Does that work?

 

[mir]

Does watchdog rely on a working ACPI?

 

[dietmar]

Does watchdog rely on a working ACPI?

no (why do you think?)

 

[mir]

If it is a software watchdog something must be responsible for rebooting the server?

 

[dietmar]

If it is a software watchdog something must be responsible for rebooting the server?

Ah, makes sense. Here is the softdog code:

https://git.kernel.org/cgit/linux/k...tchdog/softdog.c?id=refs/tags/v2.6.32.67#n106

not sure if that needs working ACPI?

 

[dietmar]

But we would at least see a log like "Initiating system reboot" - but that is also missing!

 

[mir]

The guilty party handling the actual reboot is this:
emergency_restart()
I expect it to be defined in reboot.h

 

[dietmar]

The guilty party handling the actual reboot is this:

But the log is before that code, and we do not even see the log message!

 

[mir]

Could it log to kern.log or daemon.log instead?

 

[dietmar]

Could it log to kern.log or daemon.log instead?

No, sylsog should contain all messages. To be sure, please can you check with

# journalctl

if there are watchdog related logs?

 

[mir]

I just tried your simple test:

root@pve:~# echo 1 >/dev/watchdog
-bash: /dev/watchdog: Device or resource busy

Is that the correct response?

 

[dietmar]

Ah, yes - seems watchdog-mux is still running. Try:

# systemctl stop watchdog-mux.service
# echo 1 >/dev/watchdog

 

[mir]

Yes, that did the trick.

 

[dietmar]

Yes, that did the trick.

And you got a node reboot?

 

[mir]

Yes. Within a few seconds the node rebooted.

 

[shafeeks]

Hi Dietmar,

The nodes that I am testing PVE HA does not have IPMI port. I am using the softdog of linux.

There is an easy way to test if the watchdog works correctly:

# echo 1 >/dev/watchdog

This should trigger a reboot within 60 seconds. Does that work?

Before unplugging the network cord of node 2, I test the echo 1 > /dev/watchdog on all the 3 nodes (node1, node2 and node3). It says on all the 3 nodes: "-bash: /dev/watchdog: Device or resource busy"
Then I unplugged the network cord of node2. The HA works as it migrates the VMs on node 2 evenly to node1 and node3. After some minutes, I plugged the network cord to node2 again. The membership quorum is finalised but it does not send the VMs back to node 2.

I execute "echo 1 > /dev/watchdog" command on node 2 and it got executed (while on node1 & node2, it says Device or resources busy).

After execution the command echo on node2 I checked the syslog and it fails with the below same error I had before

Aug 6 14:39:53 node2 kernel: [ 1320.827845] watchdog watchdog0: watchdog did not stop!
Aug 6 14:39:57 node2 pve-ha-lrm[1171]: watchdog update failed - Broken pipe

and it keeps repeating the last line "Broken pipe" on node2. It does NOT trigger the reboot within 60 seconds.

Each time I execute the command echo 1 > /dev/watchdog, it repeats this error on syslog.

Is there any parameters to be done on the BIOS

Thanks

Shafeek

 

[mir]

As Dietmar wrote above you might need to do this before:
# systemctl stop watchdog-mux.service
and then
# echo 1 > /dev/watchdog

As your question to fallback.
If you prefer nodes make a fallback to a previous node then you need to configure failover domains. See https://access.redhat.com/documenta...tration/s1-config-failover-domain-cli-CA.html

 

[shafeeks]

Hi Mir,

As Dietmar wrote above you might need to do this before:
# systemctl stop watchdog-mux.service
and then
# echo 1 > /dev/watchdog

Thanks for this reply. Stopping the watchdog-mux first does not change anything. It ends up with the same error as previous

Thanks also for the fallback I will check it.

A+

Shafeek

 

[dietmar]

It does NOT trigger the reboot within 60 seconds.

Maybe you manually set a kernel option for softdog (to disable reboot)?

 

[dietmar]

If you prefer nodes make a fallback to a previous node then you need to configure failover domains. See https://access.redhat.com/documenta...tration/s1-config-failover-domain-cli-CA.html

@mir: we do not use rgmanager anymore. But we have a similar feature (ha/groups).