Which backup file-type has the least variances between backups?

[Lonnie]

Which backup file-type has the least variances between backups?

For example, lets say I have a 200GB virtual machine that I backup today using zst and I move that file to a remote location.

Tomorrow, If I backup this same virtual machine again, using zst, and then use rsync (--fuzzy) to sync this zst file to that remote location (using yesterday's zst file as the fuzzy basis), will rsync be able to transfer only the differences between yesterday and today's virtual machine?

I know that rsync will only transfer the differences between the two zst files, but what I mean is, does zst produce backup files that are similar enough, between backups, that rsync can efficiently transfer ONLY the difference between yesterday's and today's virtual machine?

Previously, I was able to sync like this in just a few hours, but lately it has been taking days and I do recall seeing zst updates in Proxmox around the time that I started not being able to do efficient syncs.

I don't know much about recent updates to zst, but I'm thinking that it may be producing backup files (each backup) that are way more different than the respective virtual machine has changed (between backups).

If so, are there other backup-output-file-types, that would be more efficient for this use case?

 

[Lonnie]

I just realized that I've asked this question before:
https://forum.proxmox.com/threads/which-proxmox-backup-compression-is-best-for-rsync-fuzzy.70975/

 

[dietmar]

Please note that Proxmox Backup Server has a feature to effectively sync backup between nodes.

 

[Lonnie]

Please note that Proxmox Backup Server has a feature to effectively sync backup between nodes.

I was just reading about this today. From what I understand, installing PBS at the remote location, alone, wouldn't lead to efficient network transfer of remote backups. There would need to be a local install of PBS, and another installation at the remote location. Then, from the remote location, I could schedule sync jobs, that I guess would transfer all that is on the local PBS to the remote PBS. Is that the way it works?

I'm reading about it here. Nothing I've read so far addresses the issue I'm discussing here. The goal is to minimize the amount of data that has to be transferred between backups synced from local to the remote location. I've read that vma creates large diffs and as far as I know, zst doesn't have an rsync-friendly mode like gzip does. And, even though Proxmox supports gzip as an output option, it doesn't provide the option of "gzip --rsyncable".

How efficiently does PBS perform remote backups, and how does it overcome the obstacles I'm encountering with rsync?

Proxmox VE alone, does a great job at backing up locally, but the files it generates aren't great for syncing remotely (but they could be easily if one of the output file-types supported an rsync-friendly mode during backup). Then, I'd be able to sync in a manner where "only the changes to the virtual machine that have happened since the last backup" get transferred to the remote location.

Update: I'm finally reading Technical Overview of PBS and noticing opportunities for efficiency.

 

[dietmar]

I was just reading about this today. From what I understand, installing PBS at the remote location, alone, wouldn't lead to efficient network transfer of remote backups. There would need to be a local install of PBS, and another installation at the remote location. Then, from the remote location, I could schedule sync jobs, that I guess would transfer all that is on the local PBS to the remote PBS. Is that the way it works?

yes

I'm reading about it here. Nothing I've read so far addresses the issue I'm discussing here.

It does efficiently sync backups - I thought that is what you want to achieve.

The goal is to minimize the amount of data that has to be transferred between backups synced from local to the remote location. I've read that vma creates large diffs and as far as I know, zst doesn't have an rsync-friendly mode like gzip does.

How efficiently does PBS perform remote backups, and how does it overcome the obstacles I'm encountering with rsync?

PBS syncs only transfers the minimal set of changed chunks - very efficient (It can detect what chunks changed from saved metadata).

Proxmox VE alone, does a great job at backing up locally, but the files it generates aren't great for syncing remotely (but they could be easily if one of the output file-types supported an rsync-friendly mode during backup).

 

[Lonnie]

yes

It does efficiently sync backups - I thought that is what you want to achieve.

PBS syncs only transfers the minimal set of changed chunks - very efficient (It can detect what chunks changed from saved metadata).

Thank you for addressing my concerns.

After reading the Technical Overview, I can see how Proxmox Backup Server is likely the most efficient option available (both locally and remotely). Proxmox to NFS local backups have to transfer the whole backup file to NFS for each backup. If I'm understanding correctly, PBS only needs to transfer differences (even on local backups). Is that correct?

Not only that, but if I'm understanding correctly, a weeks worth of backups in PBS will typically take up much less space on the hard drive than would a NFS store of those same weeks worth of backups. It essentially allows Proxmox VE to support incremental backups (I guess), which is not an option for NFS storage. That would be much less wear and tear on storage and increase the life of the drives.

Then, performing remote syncs at the changed-chunk-level sounds super efficient too.

It seems like you guys have produce another fine solution. I intend to try it out soon.