Copy backups regularly to externeal usb disk

[azmosult]

Hi,

I wan't to copy pbs Backups in a regularly basis to an usb disk, or multiple usb disks.
Let's say 7 usb disks and every day of week one of this disks is plugged in into the pbs.
If a usb disk is plugged in the sync to the disk starts and the disc gets unmounted and can be unplugged.
Encryption will be done via LUKS (every linux can open it without to much hassle)

This is relatively easy to script but what would be a good way to store the data on the usb disk?

1.) Mount the latest (verified) backup of every VM and rsync it to the usb disk.
Pro: Easy to use, only differences will be transfered (rsync -> fast), getting single files from the usb disk is easy, only standard linux tools needed for accessing the backups
Con: Complete restore of an VM is harder (Linux -> relatively easy, Windows -> do not know how to restor only from files),

2.) Transfer the "drive-*.img" and the "qemu-server.conf" of the latest (verified) backup of every vm to the usb disk
Pro: Complete restore of an VM is easy, relatively easy to get single files (loop mount the *.img), only standard linux tools needed for accessing the backups
Con: The complete image needs to be transfered every time (this will take way to long on xTB of data -> yes rsync works blockwise but is super slow on big files)

3.) Use the proxmox backup server sync methode
Pro: Easy to import on pbs (mount and config change), ?Only diffs will be transfered (not sure about this)?
Con: I lack on know how, how to script this (but can learn it), not sure if I can only sync the the last (verified) backup of an VM (from the gui it looks like it syncs always the complete store -> not enough space on the usb disk for this), scripting will be more complicated, I need the pbs tools to get backups/files from the disk (I can not simply mount the disk on any linux pc and restore files)


Has someone other ideas how to store the backup data on the usb disk?
I think I would prefere point 2 because I can use it intependend from pbs (tools), but the transfer time is a blocker (and I currently do not know how to get the *.img files on the console)

 

[Dunuin]

3.) Use the proxmox backup server sync methode
Pro: Easy to import on pbs (mount and config change), ?Only diffs will be transfered (not sure about this)?
Con: I lack on know how, how to script this (but can learn it), not sure if I can only sync the the last (verified) backup of an VM (from the gui it looks like it syncs always the complete store -> not enough space on the usb disk for this), scripting will be more complicated, I need the pbs tools to get backups/files from the disk (I can not simply mount the disk on any linux pc and restore files)

An idea I didn't have tested yet:
You could make use of namespaces. Lets say you got a big datastore on internal disks "MainStore" with a namespace "Last" and a namespace "All". You then do backups to MainStore/Last and set a hourly prune job on the PBS for MainStore/Last with keep-hourly=24 so this namespace will only keep the last 24 hourly backup snapshots.
You then set up a local sync task that triggers every hour and pulls all backup snapshots from MainStore/Last to MainStore/All. This won't consume additional space because of deduplication. For this MainStore/All namespace you could set up a prune job that runs hourly and keeps the backup snapshots for a long time (keep-hourly=24, daily=14, weekly=8, monthy=12, yearly=10 or whatever you like.

Then you set up each external Disk as a new datastore (MonStore, TueStore, WedStore, ...) and create a local sync task without a schedule so it will never trigger automatically. This will then pull all the backup snapshots (so only the last 24 hourly backups) from MainStore/Last to for example MonStore.

Each day you then:
-plug in that external disk
-mount it
-disable maintaince mode for the MonStore/TueStore/...
-run a manual prune task
-run a manual GC task (it won't delete yesterdays backup snapshots, only the one from the day before yesterday)
-run a fstrim
-run that sync task manually
-after everything has finished enable maintaince mode for that datastore
-unmount disk
-unplug external disk

Would be a bit of work but you could script all that using the CLI and API...except for pluggin in/removing the disks.