Proxmox HA rebooted all my nodes...

D

d2600hz

Member
Mar 12, 2024
26
6
8
So I had a bit of an issue yesterday. We have 10 nodes in each of our clusters, and one of the clusters had an issue where we couldn't restart services and it was creating defunct processes.

I had two nodes not impacted by this, so I thought I'd migrate services to those nodes, and reboot each node one by one.

HA is enabled, affinity is running and split services between AMD/Intel hosts but everything is a single cluster.

But as I was getting onto my second node, all my nodes showed this:

Code: 
Aug 09 22:11:52 pve013-ewr watchdog-mux[1804]: client (PID 2696) watchdog is about to expire
Aug 09 22:11:52 pve013-ewr watchdog-mux[1804]: client (PID 2664) watchdog is about to expire
Aug 09 22:11:52 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:11:52 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:12:02 pve013-ewr watchdog-mux[1804]: client (PID 2696) watchdog expired - disable watchdog updates
Aug 09 22:12:02 pve013-ewr watchdog-mux[1804]: client (PID 2664) watchdog expired - disable watchdog updates
Aug 09 22:12:03 pve013-ewr watchdog-mux[1804]: exit watchdog-mux with active connections
Aug 09 22:12:03 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:12:03 pve013-ewr kernel: watchdog: watchdog0: watchdog did not stop!

This happened on every node according to journalctl.

But this event:

Code: 
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Sync members[5]: 1 4 6 7 10
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Sync left[4]: 2 5 8 9
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [TOTEM ] A new membership (1.47a) was formed. Members left: 2 5 8 9
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [TOTEM ] Failed to receive the leave message. failed: 2 5 8 9
Aug 09 22:11:49 pve013-ewr pmxcfs[2044]: [dcdb] notice: members: 1/1799, 4/1779, 6/2036, 7/2044, 10/2293
Aug 09 22:11:49 pve013-ewr pmxcfs[2044]: [status] notice: members: 1/1799, 4/1779, 6/2036, 7/2044, 10/2293
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Members[5]: 1 4 6 7 10
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [MAIN  ] Completed service synchronization, ready to provide service.

Only happened on two of the nodes but yet every single node rebooted.

Trying to understand how this happened as I can't find anything in a log that's a smoking bullet. And how we can increase visibility for introspection in future.

I've only been able to say I know what happened, but not why :D
 
Do you have a separate interface dedicated for corosync?

If possible share
/etc/network/interfaces
Corosync.conf
 
  • Like
Reactions: UdoB
d2600hz said:
I've only been able to say I know what happened, but not why :D
Click to expand...
I would expect to find some more of those QUORUM/TOTEM messages in the older logs. Maybe corosync was degraded a long time before - and nobody noticed.
d2600hz said:
And how we can increase visibility for introspection in future.
Click to expand...
Make your monitoring solution watch the output of

Code: 
~# corosync-cfgtool  -s
Local node ID 11, transport knet
LINK ID 0 udp
        addr    = 10.3.16.14
        status:
                nodeid:          1:     disconnected
                nodeid:          2:     connected
                nodeid:          4:     connected
...
and let it trigger an alarm when there is any "disconnected" state.

Make sure to have at least two corosync-rings, on (at least) two independent wires (not VLANs!), one dedicated for corosync (if possible).
 
Last edited:
longer said:
Do you have a separate interface dedicated for corosync?

If possible share
/etc/network/interfaces
Corosync.conf
Click to expand...
Yes there's a dedicated network for corosync / migration, we use bond1 as link 0.

Code: 
auto bond0
iface bond0 inet manual
        bond-slaves ens3f0np0 ens3f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        bond-lacp-rate 1

auto bond1
iface bond1 inet static
        address 10.10.250.15/24
        bond-slaves ens10f0np0 ens10f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 8960

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.108
iface vmbr0.108 inet static
        address 10.10.8.115/24
        gateway 10.10.8.254
        post-up ip route add 10.0.0.0/8 via 10.10.8.1

source /etc/network/interfaces.d/*

And corosync:


Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve001-ewr
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.10.250.1
    ring1_addr: 10.10.8.101
  }
  node {
    name: pve002-ewr
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.250.2
    ring1_addr: 10.10.8.102
  }
  node {
    name: pve003-ewr
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.250.3
    ring1_addr: 10.10.8.103
  }
  node {
    name: pve004-ewr
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.250.4
    ring1_addr: 10.10.8.104
  }
  node {
    name: pve011-ewr
    nodeid: 5
    quorum_votes: 1
    ring0_addr: 10.10.250.11
    ring1_addr: 10.10.8.111
  }
  node {
    name: pve012-ewr
    nodeid: 6
    quorum_votes: 1
    ring0_addr: 10.10.250.12
    ring1_addr: 10.10.8.112
  }
  node {
    name: pve013-ewr
    nodeid: 7
    quorum_votes: 1
    ring0_addr: 10.10.250.13
    ring1_addr: 10.10.8.113
  }
  node {
    name: pve014-ewr
    nodeid: 8
    quorum_votes: 1
    ring0_addr: 10.10.250.14
    ring1_addr: 10.10.8.114
  }
  node {
    name: pve015-ewr
    nodeid: 9
    quorum_votes: 1
    ring0_addr: 10.10.250.15
    ring1_addr: 10.10.8.115
  }
  node {
    name: pve091-ewr
    nodeid: 10
    quorum_votes: 1
    ring0_addr: 10.10.250.91
    ring1_addr: 10.10.8.191
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: EWR
  config_version: 12
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Along with a cfgtool -n


Code: 
nodeid: 1 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.2) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.102) enabled connected mtu: 1397

nodeid: 2 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.3) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.103) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.4) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.104) enabled connected mtu: 1397

nodeid: 4 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.1) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.101) enabled connected mtu: 1397

nodeid: 5 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.11) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.111) enabled connected mtu: 1397

nodeid: 6 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.12) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.112) enabled connected mtu: 1397

nodeid: 7 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.13) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.113) enabled connected mtu: 1397

nodeid: 8 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.14) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.114) enabled connected mtu: 1397

nodeid: 10 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.91) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.191) enabled connected mtu: 1397
 
  • Like
Reactions: atatury
UdoB said:
I would expect to find some more of those QUORUM/TOTEM messages in the older logs. Maybe corosync was degraded a long time before - and nobody noticed.

Make your monitoring solution watch the output of

Code: 
~# corosync-cfgtool  -s
Local node ID 11, transport knet
LINK ID 0 udp
        addr    = 10.3.16.14
        status:
                nodeid:          1:     disconnected
                nodeid:          2:     connected
                nodeid:          4:     connected
...
and let it trigger an alarm when there is any "disconnected" state.

Make sure to have at least two corosync-rings, on (at least) two independent wires (not VLANs!), one dedicated for corosync (if possible).
Click to expand...
Yes two rings, as per my previous reply to @longer
 
  • Like
Reactions: UdoB
Next question is why the split happened. Check whether ring 0 was already marginal for days:

journalctl -u corosync --since "14 days ago" | grep -Ei 'retransmit|token|link.*down|Sync left|new membership'
 
  • Like
Reactions: UdoB
JohnnieKF said:
Next question is why the split happened. Check whether ring 0 was already marginal for days:

journalctl -u corosync --since "14 days ago" | grep -Ei 'retransmit|token|link.*down|Sync left|new membership'
Click to expand...
I'm not sure what I'm looking for - nothing seems to be out whack, most of the stuff is related to the outage itself, and maintenance of hosts (updates / reboots etc)

It did get very weird during the outage, in that we saw about half the nodes disappear and then watchdog triggered.

But attached the output for perusal.
 

Attachments

d2600hz said:
I'm not sure what I'm looking for - nothing seems to be out whack, most of the stuff is related to the outage itself, and maintenance of hosts (updates / reboots etc)

It did get very weird during the outage, in that we saw about half the nodes disappear and then watchdog triggered.

But attached the output for perusal.
Click to expand...
Looks like your reboot removed one more votes at the same time the cluster was already shedding nodes.

Next thing I'd check is the host-side cause of those stalls:

journalctl -k --since "2026-08-09 21:30" --until "2026-08-09 22:15" | grep -iE 'hung task|blocked for more than|out of memory|oom|soft lockup|rcu_sched|rcu_preempt|nfs|iscsi|multipath'
 
JohnnieKF said:
Looks like your reboot removed one more votes at the same time the cluster was already shedding nodes.

Next thing I'd check is the host-side cause of those stalls:

journalctl -k --since "2026-08-09 21:30" --until "2026-08-09 22:15" | grep -iE 'hung task|blocked for more than|out of memory|oom|soft lockup|rcu_sched|rcu_preempt|nfs|iscsi|multipath'
Click to expand...
Nothing there, just the boot message from the bnxt_en module

Code: 
Aug 09 22:10:53 pve002-ewr kernel: bnxt_en 0000:4b:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog (kernel default 120s)

That's on all the nodes after the reboot. But typical kernel message.
 
d2600hz said:
Yes there's a dedicated network for corosync / migration, we use bond1 as link 0.
Click to expand...

I am not exactly sure if this the current issue you observed, but since bonding is used, the following recommendation from Proxmox documentation applies:
IEEE 802.3ad (LACP): If LACP bonds are used for corosync traffic, we strongly recommend setting bond-lacp-rate fast on the Proxmox VE node and the switch! With the default setting bond-lacp-rate slow, this mode is known to be problematic in certain failure scenarios, see below for details.
...




...with default settings,LACPDUs are only sent every 30 seconds, yielding a failover time of 90 seconds.This is too long, as nodes with HA resources will fence themselves alreadyafter roughly one minute without a stable quorum. If LACP bonds are used forcorosync traffic, we recommend setting bond-lacp-rate fast on the Proxmox VEnode and the switch! Setting this option on one side requests the other sideto send an LACPDU every second. Setting this option on both sides can reduce thefailover time in the scenario above to 3 seconds and thus prevent fencing.
Click to expand...


See 5.8.2. Corosync Over Bonds:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_corosync_over_bonds
 
  • Like
Reactions: leesteken
You must log in or register to reply here.
Top Bottom
Back