Hi,
A few days ago we had strange case – all three nodes restarted nearly at the same time, with just a few seconds difference.
Around that time our provider had a DDos attack, that did not affect our cluster, but some servers in the same VLAN were affected.
The DDos started at around 3:40, our servers rebooted around 4:42. (A second reboot happened again around 9:50, also as a consequence of a DDos attack on a negihboring server. Our servers had no big traffic.)
What I assume is that network might have been congested - our servers are connected through a redundant port-channel connection.
In our servers, to our best knowledge there is no HW watchdog, and the only thing we suspect could be the cause is the soft watchdog.
Attaching the logs from the three servers. 00,01 and 02.
What I see is that node00 has some pmxcfs issues since 2:10. This is the time when the backups are executed. Node02 is backed up to node00 between 2:00 and 2:10 (small backup, approx. 30 GB in total, over a redundant 2x1gbit port-channel network).
We are trying to find the cause of the reboots. Can it be the soft watchdog, even if there are no log entries?
I see that watchdog is being started:
syslog.1:Sep 4 04:44:13 hostname00 watchdog-mux[2747]: Watchdog driver 'Software Watchdog', version 0
syslog.1:Sep 4 04:44:13 hostname00 kernel: [ 0.075567] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
syslog.1:Sep 4 04:49:27 hostname00 pve-ha-lrm[3128]: watchdog active
Any help would be appreciated.
Thanks!
A few days ago we had strange case – all three nodes restarted nearly at the same time, with just a few seconds difference.
Around that time our provider had a DDos attack, that did not affect our cluster, but some servers in the same VLAN were affected.
The DDos started at around 3:40, our servers rebooted around 4:42. (A second reboot happened again around 9:50, also as a consequence of a DDos attack on a negihboring server. Our servers had no big traffic.)
What I assume is that network might have been congested - our servers are connected through a redundant port-channel connection.
In our servers, to our best knowledge there is no HW watchdog, and the only thing we suspect could be the cause is the soft watchdog.
Attaching the logs from the three servers. 00,01 and 02.
What I see is that node00 has some pmxcfs issues since 2:10. This is the time when the backups are executed. Node02 is backed up to node00 between 2:00 and 2:10 (small backup, approx. 30 GB in total, over a redundant 2x1gbit port-channel network).
We are trying to find the cause of the reboots. Can it be the soft watchdog, even if there are no log entries?
I see that watchdog is being started:
syslog.1:Sep 4 04:44:13 hostname00 watchdog-mux[2747]: Watchdog driver 'Software Watchdog', version 0
syslog.1:Sep 4 04:44:13 hostname00 kernel: [ 0.075567] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
syslog.1:Sep 4 04:49:27 hostname00 pve-ha-lrm[3128]: watchdog active
Any help would be appreciated.
Thanks!
