Proxmox HA rebooted all my nodes...

D

d2600hz

Member
Mar 12, 2024
29
7
8
So I had a bit of an issue yesterday. We have 10 nodes in each of our clusters, and one of the clusters had an issue where we couldn't restart services and it was creating defunct processes.

I had two nodes not impacted by this, so I thought I'd migrate services to those nodes, and reboot each node one by one.

HA is enabled, affinity is running and split services between AMD/Intel hosts but everything is a single cluster.

But as I was getting onto my second node, all my nodes showed this:

Code: 
Aug 09 22:11:52 pve013-ewr watchdog-mux[1804]: client (PID 2696) watchdog is about to expire
Aug 09 22:11:52 pve013-ewr watchdog-mux[1804]: client (PID 2664) watchdog is about to expire
Aug 09 22:11:52 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:11:52 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:12:02 pve013-ewr watchdog-mux[1804]: client (PID 2696) watchdog expired - disable watchdog updates
Aug 09 22:12:02 pve013-ewr watchdog-mux[1804]: client (PID 2664) watchdog expired - disable watchdog updates
Aug 09 22:12:03 pve013-ewr watchdog-mux[1804]: exit watchdog-mux with active connections
Aug 09 22:12:03 pve013-ewr systemd-journald[1011]: Received client request to sync journal.
Aug 09 22:12:03 pve013-ewr kernel: watchdog: watchdog0: watchdog did not stop!

This happened on every node according to journalctl.

But this event:

Code: 
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Sync members[5]: 1 4 6 7 10
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Sync left[4]: 2 5 8 9
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [TOTEM ] A new membership (1.47a) was formed. Members left: 2 5 8 9
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [TOTEM ] Failed to receive the leave message. failed: 2 5 8 9
Aug 09 22:11:49 pve013-ewr pmxcfs[2044]: [dcdb] notice: members: 1/1799, 4/1779, 6/2036, 7/2044, 10/2293
Aug 09 22:11:49 pve013-ewr pmxcfs[2044]: [status] notice: members: 1/1799, 4/1779, 6/2036, 7/2044, 10/2293
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [QUORUM] Members[5]: 1 4 6 7 10
Aug 09 22:11:49 pve013-ewr corosync[2206]:   [MAIN  ] Completed service synchronization, ready to provide service.

Only happened on two of the nodes but yet every single node rebooted.

Trying to understand how this happened as I can't find anything in a log that's a smoking bullet. And how we can increase visibility for introspection in future.

I've only been able to say I know what happened, but not why :D
 
d2600hz said:
I've only been able to say I know what happened, but not why :D
Click to expand...
I would expect to find some more of those QUORUM/TOTEM messages in the older logs. Maybe corosync was degraded a long time before - and nobody noticed.
d2600hz said:
And how we can increase visibility for introspection in future.
Click to expand...
Make your monitoring solution watch the output of

Code: 
~# corosync-cfgtool  -s
Local node ID 11, transport knet
LINK ID 0 udp
        addr    = 10.3.16.14
        status:
                nodeid:          1:     disconnected
                nodeid:          2:     connected
                nodeid:          4:     connected
...
and let it trigger an alarm when there is any "disconnected" state.

Make sure to have at least two corosync-rings, on (at least) two independent wires (not VLANs!), one dedicated for corosync (if possible).
 
Last edited:
longer said:
Do you have a separate interface dedicated for corosync?

If possible share
/etc/network/interfaces
Corosync.conf
Click to expand...
Yes there's a dedicated network for corosync / migration, we use bond1 as link 0.

Code: 
auto bond0
iface bond0 inet manual
        bond-slaves ens3f0np0 ens3f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        bond-lacp-rate 1

auto bond1
iface bond1 inet static
        address 10.10.250.15/24
        bond-slaves ens10f0np0 ens10f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 8960

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.108
iface vmbr0.108 inet static
        address 10.10.8.115/24
        gateway 10.10.8.254
        post-up ip route add 10.0.0.0/8 via 10.10.8.1

source /etc/network/interfaces.d/*

And corosync:


Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve001-ewr
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.10.250.1
    ring1_addr: 10.10.8.101
  }
  node {
    name: pve002-ewr
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.250.2
    ring1_addr: 10.10.8.102
  }
  node {
    name: pve003-ewr
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.250.3
    ring1_addr: 10.10.8.103
  }
  node {
    name: pve004-ewr
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.250.4
    ring1_addr: 10.10.8.104
  }
  node {
    name: pve011-ewr
    nodeid: 5
    quorum_votes: 1
    ring0_addr: 10.10.250.11
    ring1_addr: 10.10.8.111
  }
  node {
    name: pve012-ewr
    nodeid: 6
    quorum_votes: 1
    ring0_addr: 10.10.250.12
    ring1_addr: 10.10.8.112
  }
  node {
    name: pve013-ewr
    nodeid: 7
    quorum_votes: 1
    ring0_addr: 10.10.250.13
    ring1_addr: 10.10.8.113
  }
  node {
    name: pve014-ewr
    nodeid: 8
    quorum_votes: 1
    ring0_addr: 10.10.250.14
    ring1_addr: 10.10.8.114
  }
  node {
    name: pve015-ewr
    nodeid: 9
    quorum_votes: 1
    ring0_addr: 10.10.250.15
    ring1_addr: 10.10.8.115
  }
  node {
    name: pve091-ewr
    nodeid: 10
    quorum_votes: 1
    ring0_addr: 10.10.250.91
    ring1_addr: 10.10.8.191
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: EWR
  config_version: 12
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Along with a cfgtool -n


Code: 
nodeid: 1 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.2) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.102) enabled connected mtu: 1397

nodeid: 2 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.3) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.103) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.4) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.104) enabled connected mtu: 1397

nodeid: 4 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.1) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.101) enabled connected mtu: 1397

nodeid: 5 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.11) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.111) enabled connected mtu: 1397

nodeid: 6 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.12) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.112) enabled connected mtu: 1397

nodeid: 7 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.13) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.113) enabled connected mtu: 1397

nodeid: 8 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.14) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.114) enabled connected mtu: 1397

nodeid: 10 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.91) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.191) enabled connected mtu: 1397
 
  • Like
Reactions: atatury
UdoB said:
I would expect to find some more of those QUORUM/TOTEM messages in the older logs. Maybe corosync was degraded a long time before - and nobody noticed.

Make your monitoring solution watch the output of

Code: 
~# corosync-cfgtool  -s
Local node ID 11, transport knet
LINK ID 0 udp
        addr    = 10.3.16.14
        status:
                nodeid:          1:     disconnected
                nodeid:          2:     connected
                nodeid:          4:     connected
...
and let it trigger an alarm when there is any "disconnected" state.

Make sure to have at least two corosync-rings, on (at least) two independent wires (not VLANs!), one dedicated for corosync (if possible).
Click to expand...
Yes two rings, as per my previous reply to @longer
 
  • Like
Reactions: UdoB
Next question is why the split happened. Check whether ring 0 was already marginal for days:

journalctl -u corosync --since "14 days ago" | grep -Ei 'retransmit|token|link.*down|Sync left|new membership'
 
  • Like
Reactions: UdoB
JohnnieKF said:
Next question is why the split happened. Check whether ring 0 was already marginal for days:

journalctl -u corosync --since "14 days ago" | grep -Ei 'retransmit|token|link.*down|Sync left|new membership'
Click to expand...
I'm not sure what I'm looking for - nothing seems to be out whack, most of the stuff is related to the outage itself, and maintenance of hosts (updates / reboots etc)

It did get very weird during the outage, in that we saw about half the nodes disappear and then watchdog triggered.

But attached the output for perusal.
 

Attachments

d2600hz said:
I'm not sure what I'm looking for - nothing seems to be out whack, most of the stuff is related to the outage itself, and maintenance of hosts (updates / reboots etc)

It did get very weird during the outage, in that we saw about half the nodes disappear and then watchdog triggered.

But attached the output for perusal.
Click to expand...
Looks like your reboot removed one more votes at the same time the cluster was already shedding nodes.

Next thing I'd check is the host-side cause of those stalls:

journalctl -k --since "2026-08-09 21:30" --until "2026-08-09 22:15" | grep -iE 'hung task|blocked for more than|out of memory|oom|soft lockup|rcu_sched|rcu_preempt|nfs|iscsi|multipath'
 
JohnnieKF said:
Looks like your reboot removed one more votes at the same time the cluster was already shedding nodes.

Next thing I'd check is the host-side cause of those stalls:

journalctl -k --since "2026-08-09 21:30" --until "2026-08-09 22:15" | grep -iE 'hung task|blocked for more than|out of memory|oom|soft lockup|rcu_sched|rcu_preempt|nfs|iscsi|multipath'
Click to expand...
Nothing there, just the boot message from the bnxt_en module

Code: 
Aug 09 22:10:53 pve002-ewr kernel: bnxt_en 0000:4b:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog (kernel default 120s)

That's on all the nodes after the reboot. But typical kernel message.
 
d2600hz said:
Yes there's a dedicated network for corosync / migration, we use bond1 as link 0.
Click to expand...

I am not exactly sure if this the current issue you observed, but since bonding is used, the following recommendation from Proxmox documentation applies:
IEEE 802.3ad (LACP): If LACP bonds are used for corosync traffic, we strongly recommend setting bond-lacp-rate fast on the Proxmox VE node and the switch! With the default setting bond-lacp-rate slow, this mode is known to be problematic in certain failure scenarios, see below for details.
...




...with default settings,LACPDUs are only sent every 30 seconds, yielding a failover time of 90 seconds.This is too long, as nodes with HA resources will fence themselves alreadyafter roughly one minute without a stable quorum. If LACP bonds are used forcorosync traffic, we recommend setting bond-lacp-rate fast on the Proxmox VEnode and the switch! Setting this option on one side requests the other sideto send an LACPDU every second. Setting this option on both sides can reduce thefailover time in the scenario above to 3 seconds and thus prevent fencing.
Click to expand...


See 5.8.2. Corosync Over Bonds:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_corosync_over_bonds
 
  • Like
Reactions: leesteken
longer said:
I am not exactly sure if this the current issue you observed, but since bonding is used, the following recommendation from Proxmox documentation applies:



See 5.8.2. Corosync Over Bonds:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_corosync_over_bonds
Click to expand...
That was the only node that didn't have fast set.

We think we had something further back that caused an issue on dbus and eventually we got a lock on the fusefs, and corosync just eventually died and rebooted everything.

We're evaluating everything atm, in terms of how to monitor this better. We're not sure what caused that dbus issue, it just happened about 4 days earlier on all nodes.
 
  • Like
Reactions: longer
wendellrodriguez said:
are you check the MTU size ?
Click to expand...
Yup:


Code: 
Local node ID 9, transport knet
nodeid: 1 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.2) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.102) enabled connected mtu: 1397

nodeid: 2 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.3) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.103) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.4) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.104) enabled connected mtu: 1397

nodeid: 4 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.1) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.101) enabled connected mtu: 1397

nodeid: 5 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.11) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.111) enabled connected mtu: 1397

nodeid: 6 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.12) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.112) enabled connected mtu: 1397

nodeid: 7 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.13) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.113) enabled connected mtu: 1397

nodeid: 8 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.14) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.114) enabled connected mtu: 1397

nodeid: 10 reachable
   LINK: 0 udp (10.10.250.15->10.10.250.91) enabled connected mtu: 8853
   LINK: 1 udp (10.10.8.115->10.10.8.191) enabled connected mtu: 1397
 
d2600hz said:
So I had a bit of an issue yesterday. We have 10 nodes in each of our clusters, and one of the clusters had an issue where we couldn't restart services and it was creating defunct processes.

I had two nodes not impacted by this, so I thought I'd migrate services to those nodes, and reboot each node one by one.
Click to expand...
You've already received good pointers on troubleshooting and should continue digging into the logs. That said, the above description is very worrisome...

You had 10-node cluster where 8 of them were failing to start services. We don't have details on when this condition started, how it was discovered, who and why attempted to restart services. Presumably, "services" include cluster operations.

You said that your plan was to migrate "services" (VMs?) - and that creates additional load on cluster communication, compute, storage, and network.

Essentially, you have stated that you had 80% diminished capacity in your cluster - the fact that all nodes were fenced is not at all surprising. If this was my environment, I would not spend time trying to determine why the final reboot occurred, but rather what happened that led to "couldn't restart services and it was creating defunct processes.", and how 8 of 10 nodes became affected by this.

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
I missed your recent update about dbus finding. That is a good lead. Nowhere in this thread (unless I missed it) did you post what version of PVE you are running. If I recall correctly, there were several fixes to dbus handling, in the recent 1-2 years.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
All 9.2.6 - we try and keep things up to date.

Our major problem has been finding the root case (which everyone wants for an outage report) and dbus is the only culprit we found. So we're probably going to have to monitor that.

No idea why it happened systematically on all nodes, bar one, at once. We have an indicator to a timesync process on dbus which we're trying to follow through on.
 
If you have not done so yet, save system journal from each host as far in the past as it allows you. At this point its probably your only lifeline to RCA. I know the hindsight is 20/20, but the best time for data collection was, unfortunately, during the outage. Just dumping system state, process state, etc to a file for later analyzes could be a difference between having RCA and not having one.

In general, some obvious things to look at: any 3rd party agents or software, how does the storage play into this (Ceph?).
Looking for any outlier in the journal across all nodes is probably your best approach right now. This could also be a good exercise for the AI tokens.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: JohnnieKF and longer
You must log in or register to reply here.
Top Bottom
Back