Support for sync to "removable" backup disks

[crazycookie]

Hi there.

I'm experimenting a bit with setting up a 3-2-1 strategy for my home lab. The setup is pretty much the following:

• a tiny PVE cluster with some nodes and one shared bigger (>20TB) storage node
• a tiny PBS node with large primary backup storage (zfs, >20TB) and a hotswap HDD used for local sync from the primary backup storage
• the single sync backup disk is installed as a directory storage

The idea is to sync the latest snapshots to the single backup disk about once a month and then unplug it and move it physically offsite. (No, backblaze or similar cloud services is NOT an option). As the HDD is plugged in at a ordinary backplane hotswaps shouldn't be an issue.

However, as soon I remove the disk the proxmox-backup-proxy process runs at about 80% CPU and is bombing the logs with errors. Restarting it via the UI does not help (its actually not restarting at all). Only way to solve this is to run a systemctl stop and systemctl start explicitly (via ssh of course).

So I wonder if "offsite" backups are only possible for tape backups, not for other removable media? I know there's a "tape simulator" out there, but it would be nice to be able to directly use any removable media (HDD are much cheaper than LTO tapes).

BTW: A workaround for a "clean" procedure like removing the backup datastore first, than removing the directory storage to finally being able to unplug the disk is not an option either. Creating new datastore setups every month is no fun at all.

 

[OsvaldoP]

Think unlock datastore in offline mode is currently in progress. Last activity in dev mailing list last week

More info see here, maybe you explain why this feature is important to you

https://bugzilla.proxmox.com/show_bug.cgi?id=3156

 

[crazycookie]

Thanks, I will have a look at that. I thought the 3-2-1 would explain enough as "1" means offsite and usually even "offline" when it comes to ransomware. Thus, any solution connected to the network (specifically the internet) is not an option.

 

[OsvaldoP]

sorry for the misunderstanding

I meant, leave your consideration in bugtracker, so all devs can see that this feature is important for many users

 

[crazycookie]

Well, I did mention it in bugtracker (no reaction there) but found this one in the mailing list:

https://lists.proxmox.com/pipermail/pbs-devel/2024-March/007965.html

Not sure, but I guess its related and could solve the problem the proxmox-backup-proxystill keeps the datastore busy while holding the .lockfile.

I guess the main problem is the sematic of "offline". Right now it seems "offline" means you cannot reach the datastore via the API, but the processes still keeps the datastore internally "online".

However, for removable media "offline" should really mean "offline" even internally, ideally even "powered off" for save removal.

 

[OsvaldoP]

for your info,
patch is in "no subscription repo"

 

[crazycookie]

Hi again. I tested everything with the latest version 3.2-2. Unmounting and ejecting/hotswaping works nice. Thanks for that.
However, remounting after I inserted the disk still doesn't work without a reboot.

Do I miss something here?

 

[VictorSTS]

for your info,
patch is in "no subscription repo"

Please, which package and exact version does include the fix?

 

[santiagobiali]

I have remote pbs nodes that take turns syncing pbs datastore every week. Once the sync is done, I set the datastore as "offline" and then remove the disks.

After, I just plug them in the server and remove the "offline" tag from the datastore and it starts syncing again.


Removal script:

#!/bin/bash

# Disable sync jobs
for job in $(proxmox-backup-manager sync-job list --output-format json | jq .[].id | sed -e 's/\"//g'); do
        proxmox-backup-manager sync-job update $job --delete=schedule
done

# While syncing, wait
while [[ -n "$(proxmox-backup-manager task list | grep -E 'orico1.*running' && echo true)" ]]; do
        echo 'PBS Syncing Datastore, checking again in 30 seconds...'
        sleep 30;
done

proxmox-backup-manager datastore update orico1-pbs2 --maintenance-mode=offline
systemctl restart proxmox-backup-proxy.service
zpool export orico1

Inserting the disk back again:

#!/bin/bash

zpool import orico1 -f
sleep 2

# Enable datastore
proxmox-backup-manager datastore update orico1-pbs2 --delete=maintenance-mode

# Enable sync jobs
for job in $(proxmox-backup-manager sync-job list --output-format json | jq .[].id | sed -e 's/\"//g'); do
        proxmox-backup-manager sync-job update $job --schedule hourly
done

systemctl restart proxmox-backup-proxy.service

 

[OsvaldoP]

Our test runs on version 3.1-5 and usb disks formated with zfs

this are my commands, runs since the hotfix is puplished nice. but currently no time to write a script with errorhandling

/sbin/zfs mount usb4t
/sbin/proxmox-backup-manager datastore update usb4t --delete maintenance-mode
/sbin/proxmox-backup-manager prune-job run usb4t
/sbin/proxmox-backup-manager garbage-collection start usb4t
/sbin/proxmox-backup-manager sync-job run usb4t
/sbin/proxmox-backup-manager verify-job run usb4t
/sbin/proxmox-backup-manager datastore update usb4t --maintenance-mode offline
/sbin/zfs unmount usb4t

 

[VictorSTS]

Our test runs on version 3.1-5

Thanks. Works here too with proxmox-backup-server: 3.2.2-1 from Enterprise repo, but it would probably work with previous versions too (up to ~3.1-5 that you are using).

In my case I've notice that after setting the datastore in offline mode, it takes like 3 minutes to close all files from the datastore, even if there are no running tasks on it nor any access to the webUI / API. I've simply set a 5 minutes sleep and it seems to properly unmount the datastore. Mounting it back works fine too.

 

[OsvaldoP]

Thanks. Works here too with proxmox-backup-server: 3.2.2-1 from Enterprise repo, but it would probably work with previous versions too (up to ~3.1-5 that you are using).

In my case I've notice that after setting the datastore in offline mode, it takes like 3 minutes to close all files from the datastore, even if there are no running tasks on it nor any access to the webUI / API. I've simply set a 5 minutes sleep and it seems to properly unmount the datastore. Mounting it back works fine too.

Sometime we also notice a delay while unmounting.
Think it's not very important by our customers, because we will wait for a successfully unmount and will send a notification after that. (So far i will find time to try my phyton knowledge)

kr
Roland