Datastore synced with Rclone broken?

while I don't think that using sshfs is a great idea, I'd like to know if, network wise, a pbs sync is as intensive as a pbs backup. After all rsync is quite good in transferring only the changes.

Johannes S said:
again why?
Click to expand...

Because it's simpler and because the distinction between the vm/lxc and data is somewhat blurry (do you consider a docker container data or part of the os?).
I don't see the problem with needing a working PBS: yes, it's possible that both the PVE machine and the PBS one break at the same time, but that's what the off-site backup is for.
 
It's possible, that doesn't make it a great idea.
Regarding PBS sync only transfers chunks not already present on the target
 
Last edited:
So it's conceptually similar to rsync and not so network intensive as a backup? In that case mounting the storage box as cifs wouldn't be so bad, would it?
 
PBS Sync to CIFS datastore over WAN works , not recommended because can be slow, but for homelab and budget it's the cheapest way.
( remember to not expose cifs over wan , use firewall if you have static wan ip or use vpn ).
Here I mount the remote cifs datastore only during backups , GC and Verify.
Tip : chattr +i the mount point to not fill up the wrong disk.
 
Regarding slowness, how much are we talking about? A normal pbs backup moves around 10GB daily (uncompressed, less than half that compressed), I don't know what a pbs sync would move (on disk, I guess the pbs to pbs protocol is quite efficient but that's now what we're talking about), my internet pipe is 750Mbps symmetrical.
I guess I'll have to get an hertzner storage box and test what's the best option (rsync/rclone or pbs sync on a cifs mounted storage).
 
pbs sync is same speed as rsync.
rsync isn't faster because PBS datastore are files in .chunks , each changed data requires news files.
The rsync delta algorithm is not used here.
Speed depends on your connection and the remote connection.
Bandwidth vary a lot with cheaps remotes (as shared by many customers)
"Pipe" cannot be filled because CIFS doesn't like latency, that's why not recommended.
PBS data is spread out across million files; that's why SSDs are recommended.
here PVE+PBS on an Eco dedicated OVH ( "best effort" 1G down / 500M up )
and Remote Datastore for PBS on NTFS shared over CIFS on residential internet connection ( 2.5G down / 1G up )
20-30mins to daily Sync 10GB-20GB
30mins-1h30 to Verify ( weekly, reverify after 30 days )
3h30 to Garbage Collection (weekly) , 390GB size datastore, 180k chunks files , 1 Windows VM , 26 snapshots, 8x dedup
 
Last edited:
  • Like
Reactions: Johannes S
_gabriel said:
remember to not expose cifs over wan , use firewall if you have static wan ip or use vpn
Click to expand...
This is not possible with Hetzner storage box though If I recall correctly. It might be possible to limit it to a hetzner vserver but then the vserver costs outweighs the price benefit of the storage box.
Using storage boxes ( or any other network share over wan ) as datastores is a stupid idea, end of story
 
  • Like
Reactions: UdoB
So, I got a storage box. I didn't bother to use it as a datastore, I just used rclone to mirror the pbs datastore.
My pbs is actually in a docker container in a pc that I turn on with wakeonlan (I make sure that all the needed verify, prune and GC jobs are executed while it is on.).
Before turning it off (with a cron job) I wait that there's no task running, put the datastore in maintenance mode, do the rclone and put back the datastore in normal mode.
I then rcloned from hertzner to a different disk, added it as a datastore to a different pbs container on a different pc and ran a verify job (unchecking the "skip verified" box).
I did it twice, once after the inital rclone and once after the nightly backup, and the verify was successful. I also downloaded a random file from one of the backups (once I added the test pbs to proxmox).
Can I be reasonably sure that the remote backup is working?

1789976513839.png
 
olivluca said:
So, I got a storage box. I didn't bother to use it as a datastore, I just used rclone to mirror the pbs datastore.
Click to expand...
Why? The whole point of this thread is, that rclone isn't relieable for this and thus not supported. Running PBS in a docker container is also not supported, so you are introducing two points of potential errors (although docker propably doesn't have anything to do with your current problem). Why are you people so stubborn to do stuff which is built to break?

If you want to sync to a storagebox ( I really wouldn't do this) you could mount it as cifs mount and use PBS internal sync feature. But a more sensible setup would be to get some S3 storage at the cheapest vendor and use PBS S3 sync feature. If you really want to use the storagebox you could use ProxmoxVE native vzdump feature for backups without PBS and rclone the vzdump-archives to the storagebox. This is, what I do as additional offsite backup while having a local PBS on an old Mini-PC and a remote PBS on a vserver.

olivluca said:
Can I be reasonably sure that the remote backup is working?
Click to expand...
Only if you verify it on a regular schedule and test restoring the backups. This is the same for each storage though.
 
  • Like
Reactions: UdoB
As per the "why" because it's easy and convenient. My doubt is if rclone is unreliable per-se or the problem of the OP was because the datastore was in use during the sync.
In other words, can I trust the "VERIFY OK" reported by pbs? IIRC the OP got errors during the verify, most probably because the chunks changed during the rclone.
 
You must log in or register to reply here.
Top Bottom
Back