Datastore synced with Rclone broken?

[olivluca]

while I don't think that using sshfs is a great idea, I'd like to know if, network wise, a pbs sync is as intensive as a pbs backup. After all rsync is quite good in transferring only the changes.

again why?

Because it's simpler and because the distinction between the vm/lxc and data is somewhat blurry (do you consider a docker container data or part of the os?).
I don't see the problem with needing a working PBS: yes, it's possible that both the PVE machine and the PBS one break at the same time, but that's what the off-site backup is for.

 

[olivluca]

btw, the storage box is also mountable with cifs

 

[olivluca]

see also https://forum.proxmox.com/threads/pbs-mounting-windows-smb-share-as-datastore.113164/post-835628
mmmh...

 

[Johannes S]

It's possible, that doesn't make it a great idea.
Regarding PBS sync only transfers chunks not already present on the target

 

[olivluca]

So it's conceptually similar to rsync and not so network intensive as a backup? In that case mounting the storage box as cifs wouldn't be so bad, would it?

 

[_gabriel]

PBS Sync to CIFS datastore over WAN works , not recommended because can be slow, but for homelab and budget it's the cheapest way.
( remember to not expose cifs over wan , use firewall if you have static wan ip or use vpn ).
Here I mount the remote cifs datastore only during backups , GC and Verify.
Tip : chattr +i the mount point to not fill up the wrong disk.

 

[olivluca]

Regarding slowness, how much are we talking about? A normal pbs backup moves around 10GB daily (uncompressed, less than half that compressed), I don't know what a pbs sync would move (on disk, I guess the pbs to pbs protocol is quite efficient but that's now what we're talking about), my internet pipe is 750Mbps symmetrical.
I guess I'll have to get an hertzner storage box and test what's the best option (rsync/rclone or pbs sync on a cifs mounted storage).

 

[_gabriel]

pbs sync is same speed as rsync.
rsync isn't faster because PBS datastore are files in .chunks , each changed data requires news files.
The rsync delta algorithm is not used here.
Speed depends on your connection and the remote connection.
Bandwidth vary a lot with cheaps remotes (as shared by many customers)
"Pipe" cannot be filled because CIFS doesn't like latency, that's why not recommended.
PBS data is spread out across million files; that's why SSDs are recommended.
here PVE+PBS on an Eco dedicated OVH ( "best effort" 1G down / 500M up )
and Remote Datastore for PBS on NTFS shared over CIFS on residential internet connection ( 2.5G down / 1G up )
20-30mins to daily Sync 10GB-20GB
30mins-1h30 to Verify ( weekly, reverify after 30 days )
3h30 to Garbage Collection (weekly) , 390GB size datastore, 180k chunks files , 1 Windows VM , 26 snapshots, 8x dedup

 

[Johannes S]

remember to not expose cifs over wan , use firewall if you have static wan ip or use vpn

This is not possible with Hetzner storage box though If I recall correctly. It might be possible to limit it to a hetzner vserver but then the vserver costs outweighs the price benefit of the storage box.
Using storage boxes ( or any other network share over wan ) as datastores is a stupid idea, end of story