auto-add new VMs to a Pool for backup.

[tcabernoch]

I would like to automatically add new VMs to a Pool.

I manage backups via Pool membership.
I would like to automatically backup new VMs.
So it would be great if I could automatically stick em in a "New VMs" Pool.

How would you do that? I've heard about hook scripts. I could imagine much too complex methods involving shell scripts.

 

[Magnus-mercer]

pvesh set /pools/{poolid} -vms {vmid}

 

[tcabernoch]

Thanks @Magnus-mercer .

I guess I'll need something that reads the set of VMs (I don't know how to do that yet.)
Enumerate them into an array.
Evaluate which do NOT have Pool membership.
Loop through that list for {vmid} and execute the pool membership change.
And a cron.

I think.

 

[tcabernoch]

Magnus got me started. I still haven't found a way to evaluate things that don't have Pool memberships, but this might be just as good.

pvesh get /cluster/backup-info/not-backed-up -output-format=json

It gives me an ugly (I've always hated JSON) list of everything without a backup.
[{"name":"testbox","type":"qemu","vmid":626}]

I've created dummy backup jobs for the two Pools that won't get backups, so they drop off this "not-backed-up" list.

So now all existing VMs are either members of Pools that have real backups or Pools that have dummy backups.
When I run this command, its going to give me a list of new VMs that haven't been added to one of the backup Pools.
And It seems pretty straightforward to turn that into a command to add them to the NewVMs Pool, where they will be automatically backed up and gathered for further disposition later.

Using this API "not-backed-up" thing to manage Pool membership, is definitely roundabout, but its totally supported.
To be clear, I still don't have a way to explicitly check for lack of Pool membership, but one-to-one linking Pools to Backup Jobs lets you use API to check Backup status and infer Pool status. Instead of building logic, I'll use this API thing.

I guess I'll have to code something. Blah.

 

[tcabernoch]

Oooo.
This is ugly, but we are getting there.
This gives me a list of VMs with the rest of the data stripped out.

pvesh get /cluster/backup-info/not-backed-up -output-format=yaml | grep vmid

A bit of text cutting, and I'll have it down to the exact data I need.
Gotta write something to cycle through the list it produces.
I'm no coder, but this is pretty darn close to done.

 

[tcabernoch]

Taken as a whole, this is automated backup group management in Proxmox.
Is this new? Nobody's built this before? I sure couldn't get much help with it ...
Here it is.

• In your PVE GUI, create role-designated Pools (UAT/Prod/etc). Create a NewVMs Pool (that should be empty).
• Add all VMs to a Pool. Everything gets Pool membership.
• All Backup Jobs must be Pool-based. Create a Backup Job for every Pool. (Just disable the Backup Job for Pools that you don't want to backup, but you must create the Job.) Make a Job for NewVMs Pool (which should be empty) too.

The script logic is thus ...

• If the script finds that you don't have a Backup, you must not be in a Pool ...
• So you get added to the NewVMs pool, where you safely get backups ...

... even though we forgot about you. And then we come along later to figure out which Pool you really need to go in.

Forgive my coding if there are glaring mistakes. I'll fix anything really stupid if you tell me.
And for you one-liner freaks out there, this code is optimized to be understood, not for minimal byte-count.

#2024 Generic Network Systems Proxmox team. Free to the world.
#This script adds VMs that are not backed up to a Pool called NewVMs.
#Create the Pool and a Backup Job for it before using this script.

#List YAML vmids not backed up then strip em w grep
notBackedUp="$(pvesh get /cluster/backup-info/not-backed-up -output-format=yaml \
    | grep vmid)"

#strip the text, collapse the line, add commas
addToPool="$(echo "$notBackedUp" \
    | cut -f4 -d" " \
    | paste -sd,)"

#execute
pvesh set /pools/NewVMs -vms $addToPool

The above code runs, I don't get errors, and it does what's expected.
You'll need a cron job and save the script somewhere.
Your mileage may vary.


---------------------------------------
Other stuff:

Error checking could be added. Or logging. Or both. Sorry, its a hack. I'll quite likely build them and add them here.

The "set /pools/" command throws errors if you are already in a pool (which shouldn't happen, but what if ..)
We have an option to override the errors and force a Pool change with "allow-move". Pool membership is exclusive, you can only be in one.
Forcing pool membership changes could lead to unintended consequences if users that are unaware of the script were to add new VMs to the cluster and do unexpected things with Pools and Backup Jobs.
On the other hand, forcing NewVMs Pool membership on not-backed-up machines may help reign in users that are not correctly provisioning new VMs and is in fact part of the overall intent here. This is an option to be considered.

--------------------------------------
References:

pvesh
https://pve.proxmox.com/pve-docs/pvesh.1.html

pvesh /cluster/backup-info/not-backed-up
https://pve.proxmox.com/pve-docs/api-viewer/index.html#/cluster/backup-info/not-backed-up

pvesh set /pool/{poolid}
https://pve.proxmox.com/pve-docs/api-viewer/index.html#/pools/{poolid}

Proxmox API
https://pve.proxmox.com/wiki/Proxmox_VE_API