[SOLVED] shutdown cluster properly

[Elleni]

Because of ha settings, I would like to ask what services would ideally be stopped, in order to issue a shutdown by an usv-triggered event. Could the bash skript look something like this?

ssh root@1.2.3.4 "<systemctl stop pve-ha-lrm> && <systemctl stop pve-ha-crm> && <shutdown -hP now>" && ssh root@1.2.3.5 "<systemctl stop pve-ha-lrm> && <systemctl stop pve-ha-crm> && <shutdown -hP now>" && shutdown -hP now

Thanks in advance for answering me, which services should be stopped to prevent HA doing HA things like trying to migrate because of server shutdown.

 

[fba]

Yes, that is the way to go.
In your example the ssh commands run one after another. Better to parallelize it. The & lets the subshell of commands run in background, so both servers receive the commands at neary same time.

for i in 4 5; do 
 ( 
   ssh root@1.2.3.${i} "systemctl stop pve-ha-lrm pve-ha-crm" &&  ssh root@1.2.3.${i} "shutdown -hP now" 
) &
done
wait

 

[Elleni]

Edit to add the final version of the script - created by AI:

#!/bin/bash

LOGDIR="/var/log/rccmd"
LOGFILE="${LOGDIR}/shutdown_rccmd.log"
NODES=(1.2.3.4 1.2.3.5)
HA_TIMEOUT=60
GRACE_SLEEP=5

# ensure log directory and file exist locally
mkdir -p "${LOGDIR}"
touch "${LOGFILE}"

log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "${LOGFILE}"
}

log "===== RCCMD shutdown sequence started ====="

for NODE in "${NODES[@]}"; do
(
  log "Connecting to ${NODE}"

  ssh root@${NODE} "
    mkdir -p ${LOGDIR} 2>/dev/null || true
    touch ${LOGFILE} 2>/dev/null || true

    echo \"\$(date '+%Y-%m-%d %H:%M:%S') - Attempting HA shutdown\" >> ${LOGFILE}

    timeout ${HA_TIMEOUT} systemctl stop pve-ha-lrm pve-ha-crm || \
      echo \"\$(date '+%Y-%m-%d %H:%M:%S') - HA stop timed out or failed\" >> ${LOGFILE}

    sleep ${GRACE_SLEEP}

    echo \"\$(date '+%Y-%m-%d %H:%M:%S') - Powering off node\" >> ${LOGFILE}
    shutdown -hP now
  "

  log "Shutdown command sent to ${NODE}"
) &
done

# wait until all remote shutdown commands have been issued
wait

log "All cluster nodes handled, shutting down local server"
shutdown -hP now

 

[SteveITS]

FWIW towards the bottom of https://forum.proxmox.com/threads/clean-shutdown-of-whole-cluster.121116/post-526462 I found the HA services wouldn't properly stop on some nodes, but I was able to shut down the VMs (with the HA services running, through the GUI so they stay stopped). So YMMV?