Hallo zusammen,
I observed something strange today while patching a Proxmox cluster. In
this cluster, two nodes are patched at a time simultaneously using tmux.
The "alpha" and "beta" nodes here are in different data centers. What I
believe to have been the master during this incident ("gamma") is located
in the same DC as "alpha", but I have no idea if this is relevant.
Some logs:
Logs from the node that I believe was the master at this point:
I was of course surprised by the fact that:
...when no action had been taken by me to change this state, and as you can
see it corrected itself only ten seconds later:
Not before return migrations to beta had started though, which in this
case luckily did not cause any issues but of course could have, had this
node been in proper maintenance and not only in patching.
Do you have any thoughts around what could be the cause of this, and how
it could be avoided in the future? Is it, as I suspect, a race condition
cause by two hosts coming up at almost the exact same time, and if so
what is the solution here? Having to patch one host at a time will make
this procedure quite annoying -- should the HA service be disabled
completely after the maintenance-induced "evacuation" has completed,
until the host is ready to re-join? This seems at best like a workaround
to a non-reliable cluster state for HA.
Thank you in advance.
I observed something strange today while patching a Proxmox cluster. In
this cluster, two nodes are patched at a time simultaneously using tmux.
The "alpha" and "beta" nodes here are in different data centers. What I
believe to have been the master during this incident ("gamma") is located
in the same DC as "alpha", but I have no idea if this is relevant.
Some logs:
Code:
10:29:29 alpha pve-ha-lrm[3529]: status change active => maintenance
10:33:15 alpha pve-ha-lrm[1168788]: status change startup => wait_for_agent_lock
10:33:21 alpha pve-ha-crm[1169191]: status change startup => wait_for_quorum
10:33:26 alpha pve-ha-crm[1169191]: status change wait_for_quorum => slave
10:36:11 alpha pve-ha-crm[2852]: status change startup => wait_for_quorum
10:36:16 alpha pve-ha-lrm[3536]: status change startup => wait_for_agent_lock
10:36:21 alpha pve-ha-crm[2852]: status change wait_for_quorum => slave
10:29:30 beta pve-ha-lrm[3520]: status change active => maintenance
10:33:13 beta pve-ha-lrm[1788116]: status change startup => wait_for_agent_lock
10:33:15 beta pve-ha-crm[1788125]: status change startup => wait_for_quorum
10:33:20 beta pve-ha-crm[1788125]: status change wait_for_quorum => slave
10:36:44 beta pve-ha-crm[2837]: status change startup => wait_for_quorum
10:36:49 beta pve-ha-lrm[3641]: status change startup => wait_for_agent_lock
10:36:54 beta pve-ha-lrm[3641]: status change wait_for_agent_lock => active
10:36:55 beta pve-ha-crm[2837]: status change wait_for_quorum => slave
10:37:04 beta pve-ha-lrm[3641]: status change active => maintenanceLogs from the node that I believe was the master at this point:
Code:
10:36:50 gamma pve-ha-crm[2856]: node 'beta': state changed from 'maintenance' => 'online'
10:36:50 gamma pve-ha-crm[2856]: moving service 'vm:2015' back to 'beta', node came back from maintenance.
10:36:50 gamma pve-ha-crm[2856]: migrate service 'vm:2015' to node 'beta' (running)
10:36:50 gamma pve-ha-crm[2856]: service 'vm:2015': state changed from 'started' to 'migrate' (node = REDACTED, target = beta)
10:36:50 gamma pve-ha-crm[2856]: moving service 'vm:2034' back to 'beta', node came back from maintenance.
10:36:50 gamma pve-ha-crm[2856]: migrate service 'vm:2034' to node 'beta' (running)
10:36:50 gamma pve-ha-crm[2856]: service 'vm:2034': state changed from 'started' to 'migrate' (node = gamma, target = beta)
10:36:50 gamma pve-ha-crm[2856]: moving service 'vm:2045' back to 'beta', node came back from maintenance.
10:36:50 gamma pve-ha-crm[2856]: migrate service 'vm:2045' to node 'beta' (running)
10:36:50 gamma pve-ha-crm[2856]: service 'vm:2045': state changed from 'started' to 'migrate' (node = REDACTED, target = beta)
10:37:00 gamma pve-ha-crm[2856]: node 'beta': state changed from 'online' => 'maintenance'I was of course surprised by the fact that:
"node 'beta': state changed from 'maintenance' => 'online'"...when no action had been taken by me to change this state, and as you can
see it corrected itself only ten seconds later:
node 'beta': state changed from 'online' => 'maintenance'Not before return migrations to beta had started though, which in this
case luckily did not cause any issues but of course could have, had this
node been in proper maintenance and not only in patching.
Do you have any thoughts around what could be the cause of this, and how
it could be avoided in the future? Is it, as I suspect, a race condition
cause by two hosts coming up at almost the exact same time, and if so
what is the solution here? Having to patch one host at a time will make
this procedure quite annoying -- should the HA service be disabled
completely after the maintenance-induced "evacuation" has completed,
until the host is ready to re-join? This seems at best like a workaround
to a non-reliable cluster state for HA.
Thank you in advance.
