Mirroring Backups to multiple storages

iMer

New Member
Feb 17, 2015
16
0
1
Hello,

Currently we have multiple backup jobs, one per storage.
This proves to be quite the performance hog though. There are constantly (during morning-noon) backup jobs running - even some timing out due to not being able to aquire the vzdump lock in time.

What is the recommended way to mirror backups to multiple storages without running multiple backup jobs?
Is there some sort of hook we can use to call a simple mirror script?
 

symmcom

Renowned Member
Oct 28, 2012
1,084
36
68
Calgary, Canada
www.symmcom.com
Do you mean you have same VM backup job running on multiple Storages? When you say "Mirroring Backups" this is what it seems you are doing right now.
If you are making a VM backup on multiple storage for backup safe keep, i think there are more efficient way to do it. I would suggest using something like rsync between backup storages. From Proxmox you only create one backup task to a single storage. Then setup rsync from that storage to rest of the backup storage. This way you will free up Proxmox load. Rsync will keep all you storages in sync.
 

iMer

New Member
Feb 17, 2015
16
0
1
Yes, that is what I am trying to do.
Is there a recommended way to place a hook so rsync gets executed once the backup is done?
 

symmcom

Renowned Member
Oct 28, 2012
1,084
36
68
Calgary, Canada
www.symmcom.com
Yes, that is what I am trying to do.
Is there a recommended way to place a hook so rsync gets executed once the backup is done?

I believe you have to execute Rsync on the main backup storage itself. Usually rsync works like every other copy format:

#rsync <localhost:local_directory> -- > <remotehost:remote_directory>

If you want to execute rsync from proxmox backup node, it would look something like this:

#rsync <remotehost:remote_directory> -- > <remotehost:remote_directory>

I never tried rsync this way so cannot comment on that for sure. Should work. You can try out without breaking anything since rsync is just a command line program. First thing to do would be create a passwordless login between nodes through sskh key transfer. This is done automatically between Proxmox nodes. What OS are you using for your backup storages?
 

iMer

New Member
Feb 17, 2015
16
0
1
I do know how to use rsync, but I need to execute rsync (or any other syncronization utility) AFTER the backups have been created.

I was asking if there was an easy/recommended way of executing a command once the backup task has been finished
 

sumsum

Active Member
Oct 26, 2009
161
2
38
Bern, Switzerland
www.dmd2.com
I do backup of the backups from the nfs backup Share with rsync. We do that in the other night after the backups done. I'm not aware of an api of Proxmox what could trigger a script immediately the Backups are done.
Rsync is pretty easy to use.
 

DerDanilo

Renowned Member
Jan 21, 2017
454
108
63
I think that the PVE WebUI Backup Task should support to execute scripts BEFORE and AFTER the backup job runs.
This would allow to adjust as necessary or even mount specific storages as required (NFS, CIFS, iSCSI or whatever).

@proxmox team, can you please add this to the backup job?

Just created two new feature requests for that:
- https://forum.proxmox.com/threads/feature-request-pre-and-post-backup-task-hooks.38601/
- https://forum.proxmox.com/threads/f...nual-execution-of-backup-tasks-via-gui.38600/
 
Last edited:

DerDanilo

Renowned Member
Jan 21, 2017
454
108
63
@tom Can you guys please add a button to the backup panel to be able to trigger backup jobs manually?

We have two jobs, one for daily and one for weekly backups. If the weekly fails we have to change the trigger time to have it run again. It would be way easier if we could just trigger it manually if required.

I think that this should be very simple to implement, since it's mostly GUI.

Thanks! :)

proxmox2-proxmox-virtual-environment-2018-06-01-16-52-27-png.7555
 

Attachments

  • proxmox2 - Proxmox Virtual Environment 2018-06-01 16-52-27.png
    proxmox2 - Proxmox Virtual Environment 2018-06-01 16-52-27.png
    66.7 KB · Views: 101

GadgetPig

Member
Apr 26, 2016
138
24
18
52
Would storage replication be an alternative option?

https://pve.proxmox.com/wiki/Storage_Replication

"Replication uses snapshots to minimize traffic sent over the network. Therefore, new data is sent only incrementally after an initial full sync. In the case of a node failure, your guest data is still available on the replicated node....Every guest can be replicated to multiple target nodes.......Each replications bandwidth can be limited, to avoid overloading a storage or server."
 

DerDanilo

Renowned Member
Jan 21, 2017
454
108
63
Would storage replication be an alternative option?

https://pve.proxmox.com/wiki/Storage_Replication

"Replication uses snapshots to minimize traffic sent over the network. Therefore, new data is sent only incrementally after an initial full sync. In the case of a node failure, your guest data is still available on the replicated node....Every guest can be replicated to multiple target nodes.......Each replications bandwidth can be limited, to avoid overloading a storage or server."
The thing is that it does not copy the instance config over. It should copy the instance config over and add something like "-replica" or "-backup" to block starting it until manually unblocked and set into productive mode.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!