[SOLVED] 3 Feature requests: Backup VMs/CTs from nodes in backup job sequentially, Shutdown PBS after Backup complete, Etherwake PBS Host before Backup Job

[codingspiderfox]

Let's say I have a 3 node PVE Cluster

Node 1 contains
* VM100
* CT101
* CT102

Node 2
* VM104
* CT103

Node 3
* VM105

Also I have a single PBS host added as storage to the cluster.

I want to create a daily backup job that backs up all VMs
To reduce system load on the backup server which has not much compute power, not much RAM etc., I want an option in the backup job that enforces that only one VM at a time is backed up, the next should only start once all writes are done for the previous VM/CT

Also, to save power I want an option to etherwake the PBS host a few minutes before the Backup Job starts
Once the backup is complete, the PBS host should automatically shutdown

 

[fiona]

Hi,
the first part is not currently possible, but you can create one backup schedule for each node and space them out time-wise.
The start/stop is already possible via backup hooks, there is an example in /usr/share/doc/pve-manager/examples/vzdump-hook-script.pl. But if you do set up one job per node, you should only disable the storage in the job-end/job-abort hook of the last one of course.

 

[codingspiderfox]

Thanks for the quick advice on the hooks.
The documentation says
"You can find an example in the documentation directory (vzdump-hook-script.pl)."

Could you please state in the docs where the documentation directory is located? A quick Google search did not give me any clue on that

 

[Dunuin]

Could you please state in the docs where the documentation directory is located? A quick Google search did not give me any clue on that

/usr/share/doc/pve-manager/examples/vzdump-hook-script.pl

 

[fiona]

Thanks for the quick advice on the hooks.
The documentation says
"You can find an example in the documentation directory (vzdump-hook-script.pl)."

Could you please state in the docs where the documentation directory is located? A quick Google search did not give me any clue on that

The path is literally there in my comment

 

[wchv]

Is there documentation on what all information is available and passed through to that hook script? Does that example script contain an exhaustive list of the data available through the hook? For example, can I get failure reasons/codes? If I want to use the hook to monitor and report backup failures, would I only need to monitor the "abort" hook? Should I just check every logfile for ERROR messages? What about backups going to PBS? I just want a way to reliably integrate backup failure reporting into my company's monitoring solution (zabbix). Preferably the solution should work for both stand alone backups and backups to proxmox backup servers.

 

[fiona]

Is there documentation on what all information is available and passed through to that hook script? Does that example script contain an exhaustive list of the data available through the hook?

Yes, I think the example script currently does contain an exhaustive list. There are also comments for some variables describing when they might not be available. You can also dump $ENV yourself to see what's there.

 

[wchv]

Yes, I think the example script currently does contain an exhaustive list. There are also comments for some variables describing when they might not be available. You can also dump $ENV yourself to see what's there.

OK, ty. A couple more questions:

If I want to use the hook to monitor and report backup failures, would I only need to monitor the "abort" hook? or am i better just catching every jobs' end hook and check for ERROR messages in the log? Since no logs available for PBS targets, how do i handle this? Is there a backup hook i can catch on the pbs server? I am looking for a way to reliably integrate backup failure reporting into my company's monitoring solution (zabbix). Preferably the solution should work for both stand alone backups and backups to proxmox backup servers.

 

[fiona]

OK, ty. A couple more questions:

If I want to use the hook to monitor and report backup failures, would I only need to monitor the "abort" hook?

Yes, for each guest backup that fails, the backup-abort hook will be called. If there's a fundamental issue with the job overall (e.g. hookscript failed for job-start), the job-abort hook will be called.

or am i better just catching every jobs' end hook and check for ERROR messages in the log? Since no logs available for PBS targets, how do i handle this? Is there a backup hook i can catch on the pbs server?

The logs for PBS are uploaded to PBS. There's still task logs on the Proxmox VE side, but using the backup-abort hook should be reliable.

I am looking for a way to reliably integrate backup failure reporting into my company's monitoring solution (zabbix). Preferably the solution should work for both stand alone backups and backups to proxmox backup servers.

 

[wchv]

Yes, for each guest backup that fails, the backup-abort hook will be called. If there's a fundamental issue with the job overall (e.g. hookscript failed for job-start), the job-abort hook will be called.


The logs for PBS are uploaded to PBS. There's still task logs on the Proxmox VE side, but using the backup-abort hook should be reliable.

Awesome! Good to know. Thank you for your response!