Hi @philkunz , got a few questions on your shutdown tool.
We have 2 nodes with 2 UPSes (SNMP enabled), so I can see you can add multiple UPS to a single node.
1) how does it handle multiple UPSes? I can see each UPS device looks to have a threshold, but then theres a group you can add UPSes to with its own thresholds. If one UPS goes on battery and the other is on power, what happens? Similarly if both UPSes go into battery mode, is thresholds based on whichever is lowest or aggregate or higher? Is this based on if they are in a group or added as standalone?
2) How does it handle Proxmox clusters? In our ESXI environment when the shutdown event occurred sometimes it would try to migrate the VMs to the other host, even though both are being asked to shutdown. If I setup HA on my VM's does your tool prevent HA migration and try to clean shutdown the VM's as quick as possible and prevent HA movement?
Regards
Damien
We have 2 nodes with 2 UPSes (SNMP enabled), so I can see you can add multiple UPS to a single node.
1) how does it handle multiple UPSes? I can see each UPS device looks to have a threshold, but then theres a group you can add UPSes to with its own thresholds. If one UPS goes on battery and the other is on power, what happens? Similarly if both UPSes go into battery mode, is thresholds based on whichever is lowest or aggregate or higher? Is this based on if they are in a group or added as standalone?
2) How does it handle Proxmox clusters? In our ESXI environment when the shutdown event occurred sometimes it would try to migrate the VMs to the other host, even though both are being asked to shutdown. If I setup HA on my VM's does your tool prevent HA migration and try to clean shutdown the VM's as quick as possible and prevent HA movement?
Regards
Damien
Last edited:
Hi Damien,
As of v5.7.0, this is how it works:
Instead, each group action evaluates each member UPS against that action’s own thresholds:
What it does:
What it still does not do:
If you think parts of this should be solved differently, feel free to start a discussion about how things should be and why. We want to make nupst better for everyone.
Regards,
Phil
As of v5.7.0, this is how it works:
1) Multiple UPSes / groups
You can attach actions either directly to individual UPS devices, or to a group.- UPS-level actions are evaluated per UPS, independently.
- Group-level actions are evaluated across all members after a full poll cycle.
Instead, each group action evaluates each member UPS against that action’s own thresholds:
- redundant
- power-change logic treats the group as “on battery” only when all member UPSes are on battery
- threshold-based actions fire only when all member UPSes are on battery and below that action’s thresholds
- nonRedundant
- power-change logic treats the group as “on battery” when any member UPS is on battery
- threshold-based actions fire when any member UPS is on battery and below that action’s thresholds
- If one UPS is on battery and the other is still on mains:
- direct UPS actions on that affected UPS can still trigger based on that UPS alone
- a redundant group action will not trigger yet
- a nonRedundant group action can trigger once that affected UPS crosses the configured threshold
- If both UPSes are on battery:
- redundant waits until both are below the group action threshold
- nonRedundant fires as soon as either one is below the group action threshold
- on the UPS = standalone/per-UPS logic
- on the group = group mode logic
- put alerting actions on the individual UPSes if you want
- put the actual Proxmox shutdown + host shutdown on the group in redundant mode
2) Proxmox clusters / HA
NUPST is still node-local, not a cluster-wide coordinator.What it does:
- shuts down the VMs and LXCs on the local node
- waits for graceful shutdown
- force-stops remaining guests if configured
- then the host shutdown action can run after that
- there is now an HA-aware mode: proxmoxHaPolicy: "haStop"
- in that mode, HA-managed guests are told to go to stopped through the Proxmox HA layer instead of only sending plain qm/pct shutdown
What it still does not do:
- no cluster-wide shutdown orchestration
- no global HA disable
- no coordination between hosts beyond “each host handles itself”
- run NUPST on both nodes
- use proxmoxHaPolicy: "haStop" on the Proxmox action
- place the Proxmox action before the host shutdown action
If you think parts of this should be solved differently, feel free to start a discussion about how things should be and why. We want to make nupst better for everyone.
Regards,
Phil
Hi @philkunz
Thanks for the very detailed answer, this helps alot in how I set the thresholds up. I was on 5.6, so I upgraded to get the HaPolicy stuff. It doesn't look like there is an edit action command on the CLI so I edited the config file and restarted the service. So a suggestion would be an action edit command? (like editing the groups & ups worked great). Apart from that the changes you have quickly added (and the tool in general) means I can use this instead of figuring out NUTs.
As for how you are handling the HA/cluster environment, It think it works fine for a small environment like ours (I can have two terminals open and repeat commands as necessary). I just need it to shutdown cleanly and quickly as possible. For large clusters I cannot provide any guidance but I'm sure they would like something sitting in the Datacenter view to control it all/automate etc. Is it feasible to copy the config file to multiple nodes, (ie the ids don't cause issues if they are duplicate? It doesn't seem to me as nupst doesn't do host to host comms afaict)
BTW, our APC Smart-UPS 3000 using the Network Management Card via SNMPv1 reports runtime in ticks, not the more common minutes as suggested by your wizard. I got surprised I had 1000's of minutes left on my UPS, did I just download more battery
? This of course may be a one off, I don't know, but it was easy to change the setup of the UPS via the CLI.
Now I just have to find downtime to test it all. >_<
Regards
Damien
Thanks for the very detailed answer, this helps alot in how I set the thresholds up. I was on 5.6, so I upgraded to get the HaPolicy stuff. It doesn't look like there is an edit action command on the CLI so I edited the config file and restarted the service. So a suggestion would be an action edit command? (like editing the groups & ups worked great). Apart from that the changes you have quickly added (and the tool in general) means I can use this instead of figuring out NUTs.
As for how you are handling the HA/cluster environment, It think it works fine for a small environment like ours (I can have two terminals open and repeat commands as necessary). I just need it to shutdown cleanly and quickly as possible. For large clusters I cannot provide any guidance but I'm sure they would like something sitting in the Datacenter view to control it all/automate etc. Is it feasible to copy the config file to multiple nodes, (ie the ids don't cause issues if they are duplicate? It doesn't seem to me as nupst doesn't do host to host comms afaict)
BTW, our APC Smart-UPS 3000 using the Network Management Card via SNMPv1 reports runtime in ticks, not the more common minutes as suggested by your wizard. I got surprised I had 1000's of minutes left on my UPS, did I just download more battery
? This of course may be a one off, I don't know, but it was easy to change the setup of the UPS via the CLI.Now I just have to find downtime to test it all. >_<
Regards
Damien
I think this is what you are looking for. I haven't tried this repo myself.
EDIT: Sorry I see you want to edit an existing action!
Last edited:
Hi @philkunz
Had a problem with your nupst tool the other day. I had twos hosts piugged into a switch which one of the UPS were plugged into, and I has to do a power cycle on the switch. nupst then said its current status of the UPS was unknown, and so shut down all our VMs. it did non however started a shutdown of the nodes.
Secondary, your website on code.foss.global cannot be reached, actually I can't even get to code.foss.global (CONNECTION_REFUSED)....
Regards
Damien
Had a problem with your nupst tool the other day. I had twos hosts piugged into a switch which one of the UPS were plugged into, and I has to do a power cycle on the switch. nupst then said its current status of the UPS was unknown, and so shut down all our VMs. it did non however started a shutdown of the nodes.
Secondary, your website on code.foss.global cannot be reached, actually I can't even get to code.foss.global (CONNECTION_REFUSED)....
Regards
Damien
@damiengm -> code.foss.global reachability: the team did some datacenter/routing/ingress work the last few days since we got some upgrades in the uplinks. Might have been this and some outdated policy whats publicly reachable and what not was live for some time. Should be fine now. We also have new monitoring coming up that will catch those issues faster.
Regarding the nupst issue: I found the cause: Proxmox guest shutdown actions could still run when the UPS became unreachable, even though host shutdown was already protected. That is fixed in nupst now.
The team also added a configurable fail-safe: you can now set an unknownMinutes threshold, so nupst will not act immediately on a switch/network outage, but can still shut down after the UPS has been unknown/unreachable for a configured number of minutes.
This is released as @serve.zone/nupst@5.12.0
Regarding the nupst issue: I found the cause: Proxmox guest shutdown actions could still run when the UPS became unreachable, even though host shutdown was already protected. That is fixed in nupst now.
The team also added a configurable fail-safe: you can now set an unknownMinutes threshold, so nupst will not act immediately on a switch/network outage, but can still shut down after the UPS has been unknown/unreachable for a configured number of minutes.
This is released as @serve.zone/nupst@5.12.0
Last edited:
Ah yes true. We had to take it offline again. Team is currently implementing some DDOS mitigation. Somehow we got discovered by multiple scrapers at the same time on the weekend, rendering everything unusable and our ddos protection was not up for the job, and the database got hammered. Will report back, once everything is back up with mitigations in place. Sorry for the inconvenience. We will also have a mirror up on gitlab.com, that will be used as fallback by any tooling like nupst.
@philkunz I am setting NUPST up on a 2 node cluster.
When we had a short poweroutage (20 seconds) NUPST did not seem to incorperate the default 5 minute delay, instead it shut-down the VM's immediatly.
The nupst status display does not seem to indicate any delay, is there some way to check if nupst indeed is aware it should wait after a status change ?
Also I realize I only have a 'Proxmox' section in my config, nothing for the localhost itself, can you post an example of hos such a two-stage config would look ? (proxmox shutdown first, then the localhost)
When we had a short poweroutage (20 seconds) NUPST did not seem to incorperate the default 5 minute delay, instead it shut-down the VM's immediatly.
The nupst status display does not seem to indicate any delay, is there some way to check if nupst indeed is aware it should wait after a status change ?
Also I realize I only have a 'Proxmox' section in my config, nothing for the localhost itself, can you post an example of hos such a two-stage config would look ? (proxmox shutdown first, then the localhost)
Hi h@rald,
thanks for the detailed report — you hit a genuine design flaw, and your post is what triggered the rewrite that shipped yesterday as nupst 6.0.0. Taking your three points in order:
1. Why the 5-minute delay wasn't applied: in nupst < 6, defaultShutdownDelay only applied to shutdown-type actions — it's the +5 in shutdown -h +5 on the host itself. Proxmox actions fired immediately on the online→onBattery transition, with no debounce and no delay. So a 20-second blip really did stop your VMs instantly. Sorry about that — it was bad default behavior, not a misconfiguration on your side.
In nupst >= 6.0.0 actions are driven by explicit conditions instead. "onBattery": { "afterMinutes": 3 } means the battery state must be sustained for 3 minutes — a short blip does nothing. And a scheduled host shutdown is now actually cancelled (shutdown -c) if power returns during the countdown; previously nothing ever cancelled it.
2. Seeing what nupst is about to do: nupst status now shows the live state of every action, e.g. PENDING (onBattery) - executes in 3m20s unless power returns, plus a banner when a host shutdown is scheduled. There's also a new nupst config explain that renders your config as a plain-language timeline, so you can verify what will happen before the next outage.
3. Two-stage config (guests first, host after):
Reading it: a real outage stops the guests at t+3m (or earlier if the battery drains below 60% / 15 min runtime), and the host halts once the battery falls below 30% — everything cancels if power returns first. If you want a visible, abortable countdown before the guests stop, add "delayMinutes": 2 to the proxmox action; it arms, counts down in nupst status, and cancels on power restore or nupst pause.
For your 2-node cluster: run one nupst per node with this config — each node watches the UPS and shuts itself down independently, so there's no coordination problem.
Upgrading: sudo nupst upgrade (or re-run the install script). One thing to know: the migration deliberately preserves your current behavior exactly — your existing proxmox action becomes "when": { "onBattery": { "afterMinutes": 0 } }, i.e. still instant, and the daemon warns about it at startup. So after upgrading, edit the action to something like the above and run nupst config explain to double-check.
thanks for the detailed report — you hit a genuine design flaw, and your post is what triggered the rewrite that shipped yesterday as nupst 6.0.0. Taking your three points in order:
1. Why the 5-minute delay wasn't applied: in nupst < 6, defaultShutdownDelay only applied to shutdown-type actions — it's the +5 in shutdown -h +5 on the host itself. Proxmox actions fired immediately on the online→onBattery transition, with no debounce and no delay. So a 20-second blip really did stop your VMs instantly. Sorry about that — it was bad default behavior, not a misconfiguration on your side.
In nupst >= 6.0.0 actions are driven by explicit conditions instead. "onBattery": { "afterMinutes": 3 } means the battery state must be sustained for 3 minutes — a short blip does nothing. And a scheduled host shutdown is now actually cancelled (shutdown -c) if power returns during the countdown; previously nothing ever cancelled it.
2. Seeing what nupst is about to do: nupst status now shows the live state of every action, e.g. PENDING (onBattery) - executes in 3m20s unless power returns, plus a banner when a host shutdown is scheduled. There's also a new nupst config explain that renders your config as a plain-language timeline, so you can verify what will happen before the next outage.
3. Two-stage config (guests first, host after):
JSON:
"actions": [
{
"id": "stop-guests",
"type": "proxmox",
"when": { "onBattery": { "afterMinutes": 3 }, "batteryBelow": 60, "runtimeBelow": 15 },
"proxmoxMode": "auto"
},
{
"id": "halt-host",
"type": "shutdown",
"when": { "batteryBelow": 30, "runtimeBelow": 10 },
"shutdownDelay": 5
}
]Reading it: a real outage stops the guests at t+3m (or earlier if the battery drains below 60% / 15 min runtime), and the host halts once the battery falls below 30% — everything cancels if power returns first. If you want a visible, abortable countdown before the guests stop, add "delayMinutes": 2 to the proxmox action; it arms, counts down in nupst status, and cancels on power restore or nupst pause.
For your 2-node cluster: run one nupst per node with this config — each node watches the UPS and shuts itself down independently, so there's no coordination problem.
Upgrading: sudo nupst upgrade (or re-run the install script). One thing to know: the migration deliberately preserves your current behavior exactly — your existing proxmox action becomes "when": { "onBattery": { "afterMinutes": 0 } }, i.e. still instant, and the daemon warns about it at startup. So after upgrading, edit the action to something like the above and run nupst config explain to double-check.