Proxmox UPS and Power Loss

I

ieronymous

Well-Known Member
Apr 1, 2019
285
21
58
45
Hi

Has someone deployed a configuration where during power loss and when the batteries level are critical-low / below a certain percentage, UPS informs proxmox and proxmox shuts down all VMs gracefully?
Does the way goes UPS->Server (idrac / ilo) ->proxmox?

Thank you
 
I installed nut on my PVE host. It can monitor a attached UPS for the power and battery status and you can then tell it to shutdown your host when only x% of battery is left. A shutdown will automatically gracefully stop your VMs.
 
You can also setup a Nut client to monitor a Nut/UPS over a network, so you can set that up inside your server VMs if you want.
 
Dunuin said:
I installed nut on my PVE host. It can monitor a attached UPS for the power and battery status and you can then tell it to shutdown your host when only x% of battery is left. A shutdown will automatically gracefully stop your VMs.
Click to expand...
The link reminded me that I knew the service but had set it aside for future testing. It all comes down if your UPS is compatible with which feature of nuc in order to be supported?

ps when you say it can monitor an attached ups, how do you mean that? Directly to the pve with usb? Connected by ethernet to the same network as the pve?
 
Last edited:
ieronymous said:
The link reminded me that I knew the service but had set it aside for future testing. It all comes down if your UPS is compatible with which feature of nuc in order to be supported?

ps when you say it can monitor an attached ups, how do you mean that? Directly to the pve with usb? Connected by ethernet to the same network as the pve?
Click to expand...
Ive got a nut server and some nut clients. My UPS is connected via USB to the nut server. The nut clients then connect to the nut server over thesnetwork so all machines will shutdown when the UPS runs low on battery.
 
No, the Clients are other physical hosts. You dont need to shutdown guests. As soon as the PVE host receives a shutdown command it will automatically use a "stop all guests" task so all guest will do a graceful shutdown and the PVE host will wait (I think 5 minutes) for them to shutdown and only if they weren't shutdown in time, the PVE host will kill the guests and shutdown itself.

You can test that without NUT by just using shutdown now while guests are running because NUT will just run a shutdown script (which you could edit) which is using this as a default.
 
Last edited:
Dunuin said:
You can test that without NUT by just using shutdown now while guests are running
Click to expand...
This is the way I am shutting a demo pve, but I don t have a way to see if he will first shutdown the VM. I assume that it does it - gracefully and not by force. If I ll try to watch via gui interface, the only thing I am going to notice is the no connection message, so i wont be able to see if the VM has been greyed out (so it is off)
 
Dunuin said:
I installed nut on my PVE host. It can monitor a attached UPS for the power and battery status and you can then tell it to shutdown your host when only x% of battery is left. A shutdown will automatically gracefully stop your VMs.
Click to expand...
Hi Dunuin, sorry for reviving this old thread, however I'm very interested in this topic. Did you install NUT server in one of Proxmox VMs? I'm just thinking do I need a separate device for NUT server or hosting it inside Proxmox that it will potentially shut down is ok?
 
Yes, just install the "nut" meta package which will install both the "nut-server" and the "nut-client" packages. The PVE node can then act as a server as well as shut itself down.
 
  • Like
Reactions: rigel.local
I have 3 nodes and one UPS serving all 3. I have installed nut and nut-snmp, i have the driver reporting the UPS ok and I am now deep in upsmon.conf.... i am now a little confused....

1. each node will spot a UPS failure at slightly different times
2. when the first node starts to shutdown i assume my HA VMs will start to migrate to another node
3. that other node is likely to start shutting down in the middle of that (i am using ceph)

Do i need to worry about this and configure a more complex script than upsmon calling /sbin/shutdown?
 
scyto said:
I have 3 nodes and one UPS serving all 3. I have installed nut and nut-snmp, i have the driver reporting the UPS ok and I am now deep in upsmon.conf.... i am now a little confused....

1. each node will spot a UPS failure at slightly different times
2. when the first node starts to shutdown i assume my HA VMs will start to migrate to another node
3. that other node is likely to start shutting down in the middle of that (i am using ceph)

Do i need to worry about this and configure a more complex script than upsmon calling /sbin/shutdown?
Click to expand...
Not sure what route you took, but I think it’s not ideal to have all the nodes separately monitoring the single UPS. Might be better to have a VM or LXC running the NUT server monitoring the UPS and then have NUT clients attached to that on all the machines you want to shutdown when the grid is down. Everything that needs to shutdown gracefully should run the NUT client and have appropriate thresholds for when to shutdown. And systems that need a little more automation can kick off what they need to kick off.
 
sienar said:
Not sure what route you took, but I think it’s not ideal to have all the nodes separately monitoring the single UPS. Might be better to have a VM or LXC running the NUT server monitoring the UPS and then have NUT clients attached to that on all the machines you want to shutdown when the grid is down. Everything that needs to shutdown gracefully should run the NUT client and have appropriate thresholds for when to shutdown. And systems that need a little more automation can kick off what they need to kick off.
Click to expand...
That part of the project has been on the back burner, the monitoring is fine, the shutdown sequencing testing was the hard part. I am confused why you think having the 3 nodes listen directly to to the UPS over the network is wrong. It make no difference if they talk to a nut server or each node talks directly to the network card in the ups. Your suggestion to add a nut server adds an additional single point of failure that isn’t needed.
 
scyto said:
That part of the project has been on the back burner, the monitoring is fine, the shutdown sequencing testing was the hard part. I am confused why you think having the 3 nodes listen directly to to the UPS over the network is wrong. It make no difference if they talk to a nut server or each node talks directly to the network card in the ups. Your suggestion to add a nut server adds an additional single point of failure that isn’t needed.
Click to expand...
The NUT server suggestion is to alleviate the timing problem you pointed out of not all the hosts catching at the same time that the UPS switched to battery. Single source of battery status that all NUT clients can take action on simultaneously.
 
sienar said:
The NUT server suggestion is to alleviate the timing problem you pointed out of not all the hosts catching at the same time that the UPS switched to battery. Single source of battery status that all NUT clients can take action on simultaneously.
Click to expand...
TBH I am unclear why I thought the snp module would cause that other than it is a periodic pull of data from the ups. I thought the nut client was same with nut server, but if nut server is a push to the nut client that sounds interesting, will take a look.
 
scyto said:
TBH I am unclear why I thought the snp module would cause that other than it is a periodic pull of data from the ups. I thought the nut client was same with nut server, but if nut server is a push to the nut client that sounds interesting, will take a look.
Click to expand...
I believe you are correct that the polling nature of the SNMP monitoring is going to cause slight difference in when each NUT server that's monitoring it would see the UPS transition but the NUT clients do maintain a connection to the NUT server and will see the transition essentially instantly. I have always done this task with a single NUT server because the UPS was monitored by serial or USB, so it could ONLY be one the NUT service that's directly connected. I've even gone so far as to put the NUT server service on a nearly dedicated device, an RPi that ran a couple other low level network services along with it. Sure, it'd be one more system to maintain but I like the idea of it being virtualized so that it can easily be moved between hosts if needed.
 
sienar said:
I believe you are correct that the polling nature of the SNMP monitoring is going to cause slight difference in when each NUT server that's monitoring it would see the UPS transition but the NUT clients do maintain a connection to the NUT server and will see the transition essentially instantly. I have always done this task with a single NUT server because the UPS was monitored by serial or USB, so it could ONLY be one the NUT service that's directly connected. I've even gone so far as to put the NUT server service on a nearly dedicated device, an RPi that ran a couple other low level network services along with it. Sure, it'd be one more system to maintain but I like the idea of it being virtualized so that it can easily be moved between hosts if needed.
Click to expand...
Yeah that’s where my thinking got stalled and oddly higher prio tasks interrupted my investigation.

My original thinking I think was how to stop unintended migrations as part of an emergency shutdown procedure. I just did some more thinking, I think I want a way to change the migration policy to freeze on all nodes until they come back up. So a way to tweak this setting https://pve.proxmox.com/wiki/High_Availability#ha_manager_shutdown_policy or maybe something from this as even with you nut server solution I can see a chance for race condition between nodes….. gonna be fun to play with.

As for a nut server, I have been thinking about a usb attached pi….. same for dns, having dns in a container always has interesting implications if cluster won’t start due do dns issues…. lol.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back