Timeout when remote syncing huge VMs

[alexschomb]

I'm trying to sync huge VM backups from a local PBS to a remote PBS (located in a datacenter), but found out that unfortunately the connection times out during the initial transfer of the first VM backup:

...
2023-11-11T00:34:59+01:00: percentage done: 25.00% (1/4 groups)
2023-11-11T00:34:59+01:00: sync snapshot vm/101/2022-12-30T17:00:18Z
2023-11-11T00:34:59+01:00: sync archive qemu-server.conf.blob
2023-11-11T00:35:00+01:00: sync archive drive-scsi1.img.fidx
2023-11-17T03:54:55+01:00: percentage done: 25.81% (1/4 groups, 1/31 snapshots in group #2)
2023-11-17T03:54:55+01:00: sync group vm/101 failed - timed out
2023-11-17T03:54:59+01:00: sync group vm/102 failed - authentication failed - invalid ticket - expired
2023-11-17T03:55:02+01:00: sync group vm/103 failed - authentication failed - invalid ticket - expired
2023-11-17T03:55:02+01:00: Finished syncing namespace remote, current progress: 3 groups, 0 snapshots
2023-11-17T03:55:02+01:00: TASK ERROR: sync failed with some errors.

The backup file of vm/101 (vm/100 is much smaller and went through smoothly) is about 6.3 TB of size and the local internet connection is limited to 40 MBit/s (Good ol' Germany). I was hoping that after the first backup is transfered successfully the following backups will be much faster (incremental). Is this even plausible and what happens if the oldest backup will be deleted because of pruning - will I still benefit of incremental backup speed?

If this is backup strategy is realistic I'm also considering to manually transfer the initial backups manually to the remote location. Do I just have to copy the files from the source and put them in the correct directory in the destination PBS? Since the remote datacenter is in a different country I would prefer the default solution of just waiting long enough.

 

[ubu]

Syncing the backup storage dir, especially .chunks should work

 

[alexschomb]

Oops, sorry for the late reply. Somehow I didn't get an email notification although I watched the thread.

Thanks for the instructions - I'll try if this works

 

[alexschomb]

By the way, it seems to work just fine. After the initial transfer of the files, an

os error 13 permission denied

could be found in the sync logs. I had to set the file/group owner manually, although it displayed the identical one in

ls -la

, to solve this:

chown -R backup:backup /backup/datastore

. Since then the sync job is running and is only transferring diffs (just a few hours per backup instead of days/weeks). Thank you!