Offsite Backups and Drive Rotation

[foxyproxy]

Hi everyone!

I've got two ZFS 8TB spinning rust drives attached to my proxmox backup server. Each drive is linked to it's own datastore. So there are two data stores.

Since both drives are in hot-swappable bays, what I'd like to do is pull both drives out, take them offsite somewhere and put two replacement drives in of the same capacity so backups can continue.

Is something like this possible? I've not tried this since I don't want to cause data corruption, but I assume it's not through the UI and I'd need to write a script which needs to be executed before I do this?

What are my alternatives if I want offsite backup? Keep in mind that the remote site doesn't have high speed internet and tape backups are out of the question due to their cost.

Any help or advice would be greatly appreciated.

 

[Chris]

HI,
this is not directly possible, as a datastore creates a well defined directory structure on creation. So your script would need to first check if there are no tasks/jobs running, disable the datastore, unmount the dataset and export the zpool, create the new zpool and dataset and recreate the directory structure before reactivating the datastore.

I would not go this route, but rather use the advantages of zfs here. You could create a zfs snapshot from the dataset you PBS datastore resides on and use zfs send/zfs recieve to sync the snapshot to a 3 drive which you than transport to your offsite location.

Also, note that if you run your zpool without redundancy, zfs cannot recover from data corruption. So it is advised to use at least a mirrored setup.

 

[foxyproxy]

Can you tell me more about zfs send and zfs receive ?

 

[Chris]

Can you tell me more about zfs send and zfs receive ?

You can find all the information regarding zfs send receive in the docs [0][1], including some practical examples.

[0] https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html
[1] https://openzfs.github.io/openzfs-docs/man/8/zfs-recv.8.html

 

[foxyproxy]

I had an idea. What if I clone drive A to another offsite drive and clone drive B to another offsite drive and then swap them around? Would backups continue if I insert the offsite drive and replace it?

I assume since the data is older that the backups would continue?

 

[greybeard]

I run PVE backup and OMV (open media vault) on the same server. My system is only lightly loaded/used and typically powered on each night for the duration of the backups to OMV. The PVE backups are basically on demand, not scheduled. OMV has a function where you can configure a backup to be initiated by connecting an external USB drive. You specify the directory to be backed up it uses rsync as the backup program.
Where I'm heading with this is, if an external USB drive is suitable as a backup, you could have multiple USB drives rotated offsite providing offsite backup. Either load OMV and use their USB backup script or copy and write your own. You would just have to pick a suitable time for it to run and then plugging in the USB drive would initiate the backup.

 

[Chris]

I had an idea. What if I clone drive A to another offsite drive and clone drive B to another offsite drive and then swap them around? Would backups continue if I insert the offsite drive and replace it?

I assume since the data is older that the backups would continue?

This could work, but maybe you elaborate on why you want to swap the disks to begin with. I thought the whole point of it was to transport the data from A to B for offsite backup without high bandwidth connection? Or do you have the restore of backups in mind, after a disk failure in your running system?

 

[foxyproxy]

The idea is to have drive rotation.

Keep in mind, I have 4 x 8TB drives in total. 2 of those drives are are in the backup server (not mirrored as I need the space). If I could take an offsite drive and swap it with a drive in the backup server then I just rotate that way and it keeps my drive count as 4.

Using your suggestion with zfs send/receive, it means I need a total of 6 drives. 2 in the live system, 2 in the live system which I use zfs send to copy to that will eventually go offsite, and 2 drives which are offsite which I swap around eventually.

So this solution requires an additional 2 drives.

What about this as an alternative idea. Is there a way I can have 4 data stores in proxmox backup server where 2 data stores are disabled and 2 are in use with the current drives. Then I take my offsite drives and then when inserting them, it disables the other 2 data stores currently in use and enables the other 2 disabled ones?

All I would need then is proxmox which changes the backup data store. Would that work?

 

[Chris]

Keep in mind, I have 4 x 8TB drives in total. 2 of those drives are are in the backup server (not mirrored as I need the space). If I could take an offsite drive and swap it with a drive in the backup server then I just rotate that way and it keeps my drive count as 4.

Using your suggestion with zfs send/receive, it means I need a total of 6 drives. 2 in the live system, 2 in the live system which I use zfs send to copy to that will eventually go offsite, and 2 drives which are offsite which I swap around eventually.

So this solution requires an additional 2 drives.

Well, you will have to pull out that drive anyway from the other machine, so why not put it in the backup server and sync directly to it? No additional drives required.

At some point you will have to write the data from one disk to the other, simply swapping disks will not achieve that, or do I misunderstand your request? The whole point of offsite backups is to have an additional copy in a different location, so just moving data from A to B is not what I would consider an offsite backup.

Note that although you stated that you are bandwidth limited, maybe an offsite backup server with a sync job pulling the latest chunks would work just fine. See https://pbs.proxmox.com/docs/managing-remotes.html

 

[foxyproxy]

No problem I'll take a look.

Thanks so much for taking the time to respond.

 

[foxyproxy]

I thought I'd share my script in case anyone else finds it useful

The script first checks to see if the script is executing by looking for a file on disk. This prevents my cronjob from executing two processes.

Second, it creates a snapshot of my datastore, named "backup-0"

The it sends the snapshot to my offsite drive, named offiste-backup-0

After it finishes, it deletes any snapshots older than 1 day

Hope it helps someone out there.

#!/bin/bash


cd /root


# Set output file
output_file="zfs-send-progress.txt"


# Check if the output file is not empty
if [[ -s $output_file ]]; then
    echo "Output file '$output_file' is not empty. Aborting script."
    exit 1
fi


# Create output file
touch $output_file


# Create a snapshot with the current date and time
echo "Creating a new snapshot..." | tee -a $output_file
zfs snapshot backup-0@$(date +"%Y-%m-%d-%H-%M")


# Get the latest snapshot
echo "Getting the latest snapshot..." | tee -a $output_file
latest_snapshot=$(zfs list -t snapshot -o name -s creation | grep 'backup-0@' | tail -n 1)


# Send the snapshot to the offsite-backup-0 pool
echo "Sending the snapshot to the offsite-backup-0 pool..." | tee -a $output_file
zfs send -v $latest_snapshot | pv | zfs receive -v offsite-backup-0/$(basename $latest_snapshot)


echo "Deleting snapshots older than 1 day from backup-0..." | tee -a $output_file
current_timestamp=$(date +%s)
one_day_seconds=$((24 * 60 * 60))


echo "Current timestamp = $current_timestamp"


zfs list -t snapshot -o name,creation -s creation | grep 'backup-0@' | while read -r snapshot_line; do
    snapshot_name=$(echo $snapshot_line | awk '{print $1}')
    echo $snapshot_name


    snapshot_timestamp=$(date -u -d "$(echo $snapshot_name | awk -F'[-@]' '{print $3"-"$4"-"$5" "$6":"$7}')" +%s)
    echo $snapshot_timestamp


    age_seconds=$((current_timestamp - snapshot_timestamp))


    if [[ $age_seconds -gt $one_day_seconds ]]; then
        echo "Deleting snapshot $snapshot_name..." | tee -a $output_file
        # zfs destroy $snapshot_name
    fi
done


# Clear output file content
rm $output_file

 

[Dunuin]

I would check if the "zfs send -v $latest_snapshot | pv | zfs receive -v offsite-backup-0/$(basename $latest_snapshot)" really finished without an error before deleting the old snapshots. Would be annoying if the replication fails and you then have to backup all data again, because incremental sends won'T work anymore, because the snapshots got out of sync.