here we do this:
for kvm use snapshot , for lxc stop mode. note after backup make sure the lxc is restarted. once in awhile I've had to start manually.
then we use rsync to send the backups to all other proxmox systems and usb for taking off site. if you want to see the script i can post later.
I have 2 Server with Proxmox by OVH.com with 1 zfs storage.
my plan!
I mount the Storage one first Server and make a full Backup from the KVM on the ZFS Storage.
Then i mount the ZFS Storage to the second Server and write the Backup back.
for pve-zsync the 1-st backup will take awhile, after that depending on changes to data in the vm it can be very fast. like seconds. at this point. pve-zsync restores can only be done from cli . in case you did not know - pve-zsync uses zfs send / receive . check pve wiki for more information.
we also use rsync so that the vzdump files are avail on all nodes. vzdump can be restored from pve web page . the disadvantage is the transfer of 60GB over network.
example using rsync
Code:
ionice -c 3 rsync --exclude=*tmp --exclude=*dat -a /pve/dump/ $TARGET:/bkup/dump/
Our actual script is a lot more complicated , it deals with getting backups from 4 nodes then sending to all nodes.
during an emergency I like having vzdump files on a local disk at any node. we use usb 3.0 to put transfer dumps to two off site locations. I am anti cloud.