Mirroring Backups to multiple storages

[iMer]

Hello,

Currently we have multiple backup jobs, one per storage.
This proves to be quite the performance hog though. There are constantly (during morning-noon) backup jobs running - even some timing out due to not being able to aquire the vzdump lock in time.

What is the recommended way to mirror backups to multiple storages without running multiple backup jobs?
Is there some sort of hook we can use to call a simple mirror script?

 

[wahmed]

Do you mean you have same VM backup job running on multiple Storages? When you say "Mirroring Backups" this is what it seems you are doing right now.
If you are making a VM backup on multiple storage for backup safe keep, i think there are more efficient way to do it. I would suggest using something like rsync between backup storages. From Proxmox you only create one backup task to a single storage. Then setup rsync from that storage to rest of the backup storage. This way you will free up Proxmox load. Rsync will keep all you storages in sync.

 

[iMer]

Yes, that is what I am trying to do.
Is there a recommended way to place a hook so rsync gets executed once the backup is done?

 

[wahmed]

Yes, that is what I am trying to do.
Is there a recommended way to place a hook so rsync gets executed once the backup is done?

I believe you have to execute Rsync on the main backup storage itself. Usually rsync works like every other copy format:

#rsync <localhost:local_directory> -- > <remotehost:remote_directory>

If you want to execute rsync from proxmox backup node, it would look something like this:

#rsync <remotehost:remote_directory> -- > <remotehost:remote_directory>

I never tried rsync this way so cannot comment on that for sure. Should work. You can try out without breaking anything since rsync is just a command line program. First thing to do would be create a passwordless login between nodes through sskh key transfer. This is done automatically between Proxmox nodes. What OS are you using for your backup storages?

 

[iMer]

I do know how to use rsync, but I need to execute rsync (or any other syncronization utility) AFTER the backups have been created.

I was asking if there was an easy/recommended way of executing a command once the backup task has been finished

 

[sumsum]

I do backup of the backups from the nfs backup Share with rsync. We do that in the other night after the backups done. I'm not aware of an api of Proxmox what could trigger a script immediately the Backups are done.
Rsync is pretty easy to use.

 

[Nemesiz]

http://www.bareos.org/en/

 

[DerDanilo]

I think that the PVE WebUI Backup Task should support to execute scripts BEFORE and AFTER the backup job runs.
This would allow to adjust as necessary or even mount specific storages as required (NFS, CIFS, iSCSI or whatever).

@proxmox team, can you please add this to the backup job?

Just created two new feature requests for that:
- https://forum.proxmox.com/threads/feature-request-pre-and-post-backup-task-hooks.38601/
- https://forum.proxmox.com/threads/f...nual-execution-of-backup-tasks-via-gui.38600/

 

[LnxBil]

Already possible by using vzdump hooks - example here:

https://github.com/proxmox/pve-manager/blob/master/vzdump-hook-script.pl

 

[DerDanilo]

Already possible by using vzdump hooks - example here:

https://github.com/proxmox/pve-manager/blob/master/vzdump-hook-script.pl

Should be more simple. Job/Tasks state handling should be done on default. User should not need to interact with this. I am thinking of mounting a specific storage right before a job or to make sure that a specific storage is really mounted.

 

[joshin]

@iMer - a dumb way to do it which works, is to have a job run that watches the backup files to see when they're done being written to, and then pushes them elsewhere.

 

[DerDanilo]

@tom Can you guys please add a button to the backup panel to be able to trigger backup jobs manually?

We have two jobs, one for daily and one for weekly backups. If the weekly fails we have to change the trigger time to have it run again. It would be way easier if we could just trigger it manually if required.

I think that this should be very simple to implement, since it's mostly GUI.

Thanks!

 

[GadgetPig]

Would storage replication be an alternative option?

https://pve.proxmox.com/wiki/Storage_Replication

"Replication uses snapshots to minimize traffic sent over the network. Therefore, new data is sent only incrementally after an initial full sync. In the case of a node failure, your guest data is still available on the replicated node....Every guest can be replicated to multiple target nodes.......Each replications bandwidth can be limited, to avoid overloading a storage or server."

 

[DerDanilo]

Would storage replication be an alternative option?

https://pve.proxmox.com/wiki/Storage_Replication

"Replication uses snapshots to minimize traffic sent over the network. Therefore, new data is sent only incrementally after an initial full sync. In the case of a node failure, your guest data is still available on the replicated node....Every guest can be replicated to multiple target nodes.......Each replications bandwidth can be limited, to avoid overloading a storage or server."

The thing is that it does not copy the instance config over. It should copy the instance config over and add something like "-replica" or "-backup" to block starting it until manually unblocked and set into productive mode.