First Sync taking much longer when doing Backup to PBS

[Tesla2k]

Hello,
I have a Proxmox VE running on a Hetzner Root Server and a PBS running local at my house. I included the PBS as Storage in the VE and doing backups to it. It is working, but the backups take very long.
Especially the Step from "starting first sync" to "first sync finished" which as far as I understand it is only local, so no slow connection can be the cause. All containers together take about 18h and the server is noticable slower during that time. So I am seeking for a solution

Here an example of a small Container. The first sync step takes 436s, when I run a backup manually to the local storage this steps takes only 36s.
/var/tmp and /var/lib/vz/dump are on the same harddisk.


Backup to PBS:

104: 2024-04-28 18:00:20 INFO: Starting Backup of VM 104 (lxc)
104: 2024-04-28 18:00:20 INFO: status = running
104: 2024-04-28 18:00:20 INFO: CT Name: <container_name>
104: 2024-04-28 18:00:20 INFO: including mount point rootfs ('/') in backup
104: 2024-04-28 18:00:20 INFO: mode failure - some volumes do not support snapshots
104: 2024-04-28 18:00:20 INFO: trying 'suspend' mode instead
104: 2024-04-28 18:00:20 INFO: backup mode: suspend
104: 2024-04-28 18:00:20 INFO: ionice priority: 7
104: 2024-04-28 18:00:20 INFO: CT Name: <container_name>
104: 2024-04-28 18:00:20 INFO: including mount point rootfs ('/') in backup
104: 2024-04-28 18:00:20 INFO: starting first sync /proc/2205099/root/ to /var/tmp/vzdumptmp2358030_104
104: 2024-04-28 18:07:36 INFO: first sync finished - transferred 3.55G bytes in 436s
104: 2024-04-28 18:07:36 INFO: suspending guest
104: 2024-04-28 18:07:36 INFO: starting final sync /proc/2205099/root/ to /var/tmp/vzdumptmp2358030_104
104: 2024-04-28 18:07:40 INFO: final sync finished - transferred 142.57M bytes in 4s
104: 2024-04-28 18:07:40 INFO: resuming guest
104: 2024-04-28 18:07:40 INFO: guest is online again after 4 seconds
104: 2024-04-28 18:07:40 INFO: creating Proxmox Backup Server archive 'ct/104/2024-04-28T16:00:20Z'
104: 2024-04-28 18:07:40 INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2358030_104/etc/vzdump/pct.conf root.pxar:/var/tmp/vzdumptmp2358030_104 --include-dev /var/tmp/vzdumptmp2358030_104/. --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 104 --backup-time 1714320020 --repository <pbshost>
104: 2024-04-28 18:07:40 INFO: Starting backup: ct/104/2024-04-28T16:00:20Z
104: 2024-04-28 18:07:40 INFO: Client name: <ve_host>
104: 2024-04-28 18:07:40 INFO: Starting backup protocol: Sun Apr 28 18:07:40 2024
104: 2024-04-28 18:07:40 INFO: Downloading previous manifest (Sat Apr 27 19:23:34 2024)
104: 2024-04-28 18:07:40 INFO: Upload config file '/var/tmp/vzdumptmp2358030_104/etc/vzdump/pct.conf' to '<pbshost>' as pct.conf.blob
104: 2024-04-28 18:07:40 INFO: Upload directory '/var/tmp/vzdumptmp2358030_104' to '<pbshost>' as root.pxar.didx
104: 2024-04-28 18:08:27 INFO: root.pxar: had to backup 955.731 MiB of 3.312 GiB (compressed 471.474 MiB) in 46.37s
104: 2024-04-28 18:08:27 INFO: root.pxar: average backup speed: 20.611 MiB/s
104: 2024-04-28 18:08:27 INFO: root.pxar: backup was done incrementally, reused 2.379 GiB (71.8%)
104: 2024-04-28 18:08:27 INFO: Uploaded backup catalog (1.515 MiB)
104: 2024-04-28 18:08:29 INFO: Duration: 48.55s
104: 2024-04-28 18:08:29 INFO: End Time: Sun Apr 28 18:08:29 2024
104: 2024-04-28 18:08:29 INFO: adding notes to backup
104: 2024-04-28 18:08:31 INFO: Finished Backup of VM 104 (00:08:11)

Local Backup:

104: 2024-04-29 09:54:37 INFO: Starting Backup of VM 104 (lxc)
104: 2024-04-29 09:54:37 INFO: status = running
104: 2024-04-29 09:54:37 INFO: CT Name: <container_name>
104: 2024-04-29 09:54:37 INFO: including mount point rootfs ('/') in backup
104: 2024-04-29 09:54:37 INFO: mode failure - some volumes do not support snapshots
104: 2024-04-29 09:54:37 INFO: trying 'suspend' mode instead
104: 2024-04-29 09:54:37 INFO: backup mode: suspend
104: 2024-04-29 09:54:37 INFO: ionice priority: 7
104: 2024-04-29 09:54:37 INFO: CT Name: <container_name>
104: 2024-04-29 09:54:37 INFO: including mount point rootfs ('/') in backup
104: 2024-04-29 09:54:37 INFO: starting first sync /proc/2205099/root/ to /var/lib/vz/dump/vzdump-lxc-104-2024_04_29-09_54_36.tmp
104: 2024-04-29 09:55:13 INFO: first sync finished - transferred 3.33G bytes in 36s
104: 2024-04-29 09:55:13 INFO: suspending guest
104: 2024-04-29 09:55:22 INFO: starting final sync /proc/2205099/root/ to /var/lib/vz/dump/vzdump-lxc-104-2024_04_29-09_54_36.tmp
104: 2024-04-29 09:55:28 INFO: final sync finished - transferred 112.92M bytes in 6s
104: 2024-04-29 09:55:28 INFO: resuming guest
104: 2024-04-29 09:55:28 INFO: guest is online again after 15 seconds
104: 2024-04-29 09:55:29 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-104-2024_04_29-09_54_36.tar.zst'
104: 2024-04-29 09:57:45 INFO: Total bytes written: 3375923200 (3.2GiB, 25MiB/s)
104: 2024-04-29 09:57:45 INFO: archive file size: 1.19GB
104: 2024-04-29 09:57:45 INFO: adding notes to backup
104: 2024-04-29 09:57:50 INFO: Finished Backup of VM 104 (00:03:14)

 

[Chris]

I have a Proxmox VE running on a Hetzner Root Server and a PBS running local at my house. I included the PBS as Storage in the VE and doing backups to it. It is working, but the backups take very long.

Hi,
bandwidth might be limited for such a case, so it is expected that the backups might take some time, however not related to the first sync as you describe.

Especially the Step from "starting first sync" to "first sync finished" which as far as I understand it is only local, so no slow connection can be the cause.

Can you verify that this is the case for every such run? Is this during a time of otherwise increased IO? Check the io delay at the time of the backup.

All containers together take about 18h and the server is noticable slower during that time. So I am seeking for a solution

Backing up LXC with PBS currently require to read all the data on the containers root filesystem and mountpoints included in the backup. This can take significant amount of time and requires IO. We are however actively working on a solution to reduce this, see https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009104.html

Here an example of a small Container. The first sync step takes 436s, when I run a backup manually to the local storage this steps takes only 36s.
/var/tmp and /var/lib/vz/dump are on the same harddisk.

Both methods, tarball backups and PBS backups use the same code path (by doing an rsync of the disk contents) to sync the running container state to the temporary directory, so this should not make a difference. Please verify that this is indeed reproducible. Are there maybe other backup tasks running in parallel?