Timing and trying to understand the process of pbr and sync.

[markf1301]

Below is an example of a pbr doing a 'sync' from a remote site. Several processes are in there.

I know that 'distance' plays into effect with the data and determines the ability of the tranfer rate. I belive I am understanding the process :
sync snapshot - process of sending the latest snapshot of data over
% done - how long/much has been done and how many snapshots are in that group. Is the 'group' the # of machines in the job - correct?
sync archive - the image file with the changed, and finally the download of the differences.

Coming from previous vendor backups - once the files are initially done, the 'sending' server will then compare what is there (locally) against the new changes (what's changed in that file) and then only send the incremental changes. pbr seems to be sending over the whole file (.blob and img.fidx) and then 'requesting' thru the download the updates. If this is correct, or how I'm understanding the process - it seems a bit backwards. By 'backwards' I am referencing the need to send the 2 files over (.blob and img.fidx) then then doing the comparisons on the remote side and updating the database there. Am I wrong? This seems inefficient as a process. I would think that the backup job would run - the differences are 'marked' and then only the updates are sent to the remote side.





2024-04-02T02:36:45-04:00: percentage done: 73.33% (14/20 groups, 4/6 snapshots in group #15)
2024-04-02T02:36:45-04:00: sync snapshot vm/241/2024-04-01T00:38:21Z
2024-04-02T02:36:45-04:00: sync archive qemu-server.conf.blob
2024-04-02T02:36:45-04:00: sync archive drive-sata0.img.fidx
2024-04-02T02:51:55-04:00: downloaded 2546384004 bytes (2.67 MiB/s)
2024-04-02T02:51:55-04:00: got backup log file "client.log.blob"
2024-04-02T02:51:55-04:00: sync snapshot vm/241/2024-04-01T00:38:21Z done
2024-04-02T02:51:55-04:00: percentage done: 74.17% (14/20 groups, 5/6 snapshots in group #15)
2024-04-02T02:51:56-04:00: sync snapshot vm/241/2024-04-02T00:40:08Z
2024-04-02T02:51:56-04:00: sync archive qemu-server.conf.blob
2024-04-02T02:51:56-04:00: sync archive drive-sata0.img.fidx
2024-04-02T03:07:37-04:00: downloaded 2637008537 bytes (2.67 MiB/s)
2024-04-02T03:07:37-04:00: got backup log file "client.log.blob"
2024-04-02T03:07:37-04:00: sync snapshot vm/241/2024-04-02T00:40:08Z done
2024-04-02T03:07:37-04:00: percentage done: 75.00% (15/20 groups)
2024-04-02T03:07:38-04:00: sync snapshot vm/300/2024-03-21T16:48:23Z
2024-04-02T03:07:38-04:00: sync archive qemu-server.conf.blob
2024-04-02T03:07:38-04:00: sync archive drive-scsi0.img.fidx
2024-04-02T03:33:05-04:00: downloaded 4280740020 bytes (2.67 MiB/s)
2024-04-02T03:33:05-04:00: got backup log file "client.log.blob"
2024-04-02T03:33:05-04:00: sync snapshot vm/300/2024-03-21T16:48:23Z done
2024-04-02T03:33:05-04:00: percentage done: 80.00% (16/20 groups)
2024-04-02T03:33:06-04:00: sync snapshot vm/301/2024-03-31T00:39:24Z
2024-04-02T03:33:06-04:00: sync archive qemu-server.conf.blob
2024-04-02T03:33:06-04:00: sync archive drive-scsi0.img.fidx
2024-04-02T05:35:23-04:00: downloaded 20358005340 bytes (2.65 MiB/s)
2024-04-02T05:35:23-04:00: got backup log file "client.log.blob"
2024-04-02T05:35:23-04:00: sync snapshot vm/301/2024-03-31T00:39:24Z done
2024-04-02T05:35:23-04:00: percentage done: 81.67% (16/20 groups, 1/3 snapshots in group #17)
2024-04-02T05:35:23-04:00: sync snapshot vm/301/2024-04-01T00:38:47Z
2024-04-02T05:35:23-04:00: sync archive qemu-server.conf.blob
2024-04-02T05:35:23-04:00: sync archive drive-scsi0.img.fidx
2024-04-02T05:36:13-04:00: downloaded 139960363 bytes (2.67 MiB/s)
2024-04-02T05:36:13-04:00: got backup log file "client.log.blob"
2024-04-02T05:36:13-04:00: sync snapshot vm/301/2024-04-01T00:38:47Z done
2024-04-02T05:36:13-04:00: percentage done: 83.33% (16/20 groups, 2/3 snapshots in group #17)
2024-04-02T05:36:13-04:00: sync snapshot vm/301/2024-04-02T00:40:39Z
2024-04-02T05:36:13-04:00: sync archive qemu-server.conf.blob
2024-04-02T05:36:14-04:00: sync archive drive-scsi0.img.fidx
2024-04-02T05:38:30-04:00: downloaded 380649085 bytes (2.68 MiB/s)
2024-04-02T05:38:30-04:00: got backup log file "client.log.blob"
2024-04-02T05:38:30-04:00: sync snapshot vm/301/2024-04-02T00:40:39Z done
2024-04-02T05:38:30-04:00: percentage done: 85.00% (17/20 groups)
2024-04-02T05:38:30-04:00: sync snapshot vm/302/2024-03-21T16:49:21Z
2024-04-02T05:38:30-04:00: sync archive qemu-server.conf.blob
2024-04-02T05:38:30-04:00: sync archive drive-ide0.img.fidx

 

[ggoller]

I know that 'distance' plays into effect with the data and determines the ability of the tranfer rate. I belive I am understanding the process :
sync snapshot - process of sending the latest snapshot of data over
% done - how long/much has been done and how many snapshots are in that group. Is the 'group' the # of machines in the job - correct?

correct, a group is a machine. https://pbs.proxmox.com/docs/terminology.html#backup-group

Coming from previous vendor backups - once the files are initially done, the 'sending' server will then compare what is there (locally) against the new changes (what's changed in that file) and then only send the incremental changes. pbr seems to be sending over the whole file (.blob and img.fidx) and then 'requesting' thru the download the updates. If this is correct, or how I'm understanding the process - it seems a bit backwards. By 'backwards' I am referencing the need to send the 2 files over (.blob and img.fidx) then then doing the comparisons on the remote side and updating the database there. Am I wrong? This seems inefficient as a process. I would think that the backup job would run - the differences are 'marked' and then only the updates are sent to the remote side.

This is not quite correct.
We pull all the snapshots from the remote, except those that are strictly older than the current one. When pulling the snapshots, we perform these steps:
- (Re)download the manifest
-- if it matches, only download log and treat snapshot as already synced
- Iterate over referenced files
-- if file already exists, verify contents
-- if not, pull it from the remote
- Download log if not already existing