Triggering LXC/VM ordered shutdown from NUT

[Dultas]

Looking for some advice in case I've missed something.

I'm looking to configure NUT to shutdown LXC/VMs that are non crucial when the notification for 'ONBATT' occur. The remaining I would shut down on 'LOWBATT'.

To do this I'm looking for a way to get a list of lxc / vm that don't have a tag on them ie "crucial". I'm aware of qm list and pct list but I'm not seeing any way of filtering them based on tags. I'd like to avoid having to parse the list of all the conf files if I can, if not it's probably not that difficult a script but having a command that could just spit out the list would be even easier.

Additionally if the power comes back 'ONLINE' I'm curious if there is a simple way to trigger the on boot ordered sequence of startups for everything that is start at boot?

Thanks for any info in advance.

 

[waltar]

# For machines that could be shutdown here is the listing but you could instead of "echo" do "pct stop"/"qm stop" at end also:
for lxc in $(pct list|grep running|awk '{print $1}');do pct config $lxc|grep tags|grep crucial >/dev/null || echo $lxc ;done
for vm in $(qm list|grep running|awk '{print $1}');do qm config $vm|grep tags|grep crucial >/dev/null || echo $vm ;done

# For machines that should start here is the listing but you could instead of "echo" do "pct start"/"qm start" at end also:
for lxc in $(pct list|awk 'FNR>1 {print $1}');do pct config $lxc|grep onboot >/dev/null && echo $lxc ;done
for vm in $(qm list|awk 'FNR>1 {print $1}');do qm config $vm|grep onboot >/dev/null && echo $vm ;done

 

[Dultas]

@waltar thanks for the reply. I was hoping for a cleaner solution but imagined awk / and grep would come into it. I'll probably need to get the list and process it further so I can shutdown in order. Should be fairly easy in perl script.

 

[waltar]

Should be fairly easier with a emergency power generator or a solar panel with batteries for your pve so you don't need scripting for power down and up later your vm's and lxc's also ... ^^

 

[Dultas]

So this hasn't been fully tested. And I'd still like to find a way to check if backups are running and cancel then if I go on battery power. I'll follow up with any edits from testing. RUN_AS_USER will need to be root or another user with permissions to run pvenode and pvesh commands.

upssched.pl

#!/usr/bin/perl

use strict;
use warnings;

#fill with environment specific values
my $admin_user = '';
my $admin_pass = '';
my $critical_tag = '';
my @upses = ('');

#get node name remove newline.
chomp(my $node_name = `hostname --short`);

print "Processing $ARGV[0]. On PVE node $node_name.\n";

#process various commands from upssched.conf
#email sent to root@localhost will be processed by the Proxmox notification system
if($ARGV[0] eq "online") {
    my $msg = "Power restored. Begining ordered restart of onboot guests.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Power restored' root\@localhost`;

    #startall will start all lxc / vm marked as on boot in configured order.
    system('pvenode startall') and die('pvenode startall failed.');
}
elsif($ARGV[0] eq "onbatt") {
    my $msg = "On battery power.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS On battery' root\@localhost`;
}
elsif($ARGV[0] eq "onbatt_noncrit") {
    my $msg = "On battery power. Begining orderd shutdown of all guests not tagged critical.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS On battery, stopping guests' root\@localhost`;

    #get list of all ids for lxc and vm that are running and do not have the critical tag set.
    my $guests = `pvesh get /nodes/$node_name/lxc --noborder 1 --noheader 1 | grep '^running' | grep -v '[ ;]${critical_tag}[ ;]' | awk '{print \$2}'`;
    $guests = $guests . `pvesh get /nodes/$node_name/qemu --noborder 1 --noheader 1 | grep '^running' | grep -v '[ ;]${critical_tag}[ ;]' | awk '{print \$2}'`;
    $guests = join(',', split(/\n/, $guests));

    if($guests ne "") {
      $msg = "Stopping LXC / VM IDs: $guests.";
      print "$msg\n";
      `echo $msg | mail -s 'UPS Stopping guests' root\@localhost`;

      #stopall when provided a list of ids will shutdown only those lxc / vm and in the order configured.
      system("pvenode stopall --vms $guests") and die("pvenode stopall --vms $guests failed.");
    }
    else {
      print "No guests to shutdown.\n";
    }
}
elsif($ARGV[0] eq "lowbatt") {
    my $msg = "On low battery power. Begining ordered shutdown of all LXC / VM.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Battery Critical' root\@localhost`;

    #stopall will stop all lxc / vm in the order they are configured
    system('pvenode stopall') and die('pvenode stopall failed.');
}
elsif($ARGV[0] eq "fsd") {
    my $msg = "Forced shutdown. Stopping all LXC / VM.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Forced Shutdown' root\@localhost`;

    #stopall will stop all lxc / vm in the order they are configured
    system('pvenode stopall') and die('pvenode stopall failed.');
}
elsif($ARGV[0] eq "commbad") {
    my $msg = "Communication lost to UPS.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Comms lost' root\@localhost`;
}
elsif($ARGV[0] eq "commok") {
    my $msg = "Communication restore to UPS.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Comms restored.' root\@localhost`;
}
elsif($ARGV[0] eq "shutdown") {
    my $msg = "Shutdown requested. Stopping all LXC / VM";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Shutdown' root\@localhost`;

    #stopall will stop all lxc / vm in the order they are configured
    system('pvenode stopall');
}
elsif($ARGV[0] eq "replbatt") {
    my $msg = "Battery testing indicates replacement needed.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS Battery need replacement.' root\@localhost`;
}
elsif($ARGV[0] eq "nocomm") {
    my $msg = "UPS is unavailable.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS is unavailable' root\@localhost`;
}
elsif($ARGV[0] eq "noparent") {
    my $msg = "upsmon parent process has died. Can not perform shutdown.";
    print "$msg\n";
    `echo $msg | mail -s 'UPS upsmon dead' root\@localhost`;
}
#if admin details are set we can mute or re-enable ups beeps
elsif($ARGV[0] eq "mute_beep") {
    if($admin_user ne "" && $admin_user ne "") {
        for(@upses) {
            print "Muting beep on $_.\n";
            `upscmd -u $admin_user -p $admin_pass $_ beeper.mute`;
        }
    }
    else {
        print "Can't mute beeper. Admin not set.\n";
    }
}
elsif($ARGV[0] eq "enable_beep") {
    if($admin_user ne "" && $admin_user ne "") {
        for(@upses) {
            print "Enablign beep on $_.\n";
            `upscmd -u $admin_user -p $admin_pass $_ beeper.enable`;
        }
    }
    else {
        print "Can't enable beeper. Admin not set.\n";
    }
}
else {
    die("Unknown event: $ARGV[0]. Fail.\n");
}

And the config
upssched.conf

CMDSCRIPT /etc/nut/upssched.pl
PIPEFN /run/nut/upssched.pipe
LOCKFN /run/nut/upssched.lock

#send notification on power state change
AT ONBATT * EXECUTE onbatt

#delay onpower in case power is flickering
AT ONLINE * START-TIMER online 10
AT ONBATT * CANCEL-TIMER online

#after 60s perform non critical shutdown
AT ONBATT * START-TIMER onbatt_noncrit 60
AT ONLINE * CANCEL-TIMER onbatt_noncrit

#mute beeping at 2 minutes, re-enable when on power
AT ONBATT * START-TIMER mute_beep 120
AT ONLINE * CANCEL-TIMER mute_beep
AT ONLINE * EXECUTE enable_beep

#on low battery stop all guests
AT LOWBATT * EXECUTE lowbatt

#forced shutdown stop all guests
AT FSD * EXECUTE fsd

#handle communication errors
AT COMMBAD * START-TIMER commbad 60
AT COMMOK * CANCEL-TIMER commbad
AT COMMOK * EXECUTE commok

#shutdown stop all guests
AT SHUTDOWN * EXECUTE shutdown

#notify of barrery needing replacement
AT REPLBATT * EXECUTE replbatt

#ups issues
AT NOCOMM * EXECUTE nocomm
AT NOPARENT * EXECUTE noparent

*edit: added comments in .conf
*edit: wrong name in .conf onpower -> online

 

[waltar]

Thanks for sharing your resulting script to all who are interested