Slow backup

[liane]

Hi,

Backup Server 2.1-2

Backups, while working ok (when not at the same time as replication), are very slow, in fact they take the same time as if they were full backups locally.

Example for one container, hosted on NodeA, replicated on NodeB, and backup to BPS hosted on nodeB.

Replication: 6s

2021-12-16 11:40:01 101-1: start replication job
2021-12-16 11:40:01 101-1: guest => CT 101, running => 1
2021-12-16 11:40:01 101-1: volumes => local-zfs:subvol-101-disk-0
2021-12-16 11:40:02 101-1: freeze guest filesystem
2021-12-16 11:40:04 101-1: create snapshot '__replicate_101-1_1639651201__' on local-zfs:subvol-101-disk-0
2021-12-16 11:40:05 101-1: thaw guest filesystem
2021-12-16 11:40:05 101-1: using secure transmission, rate limit: 100 MByte/s
2021-12-16 11:40:05 101-1: incremental sync 'local-zfs:subvol-101-disk-0' (__replicate_101-1_1639649401__ => __replicate_101-1_1639651201__)
2021-12-16 11:40:05 101-1: using a bandwidth limit of 100000000 bps for transferring 'local-zfs:subvol-101-disk-0'
2021-12-16 11:40:05 101-1: send from @__replicate_101-1_1639649401__ to rpool/data/subvol-101-disk-0@__replicate_101-1_1639651201__ estimated size is 28.2M
2021-12-16 11:40:05 101-1: total estimated size is 28.2M
2021-12-16 11:40:07 101-1: successfully imported 'local-zfs:subvol-101-disk-0'
2021-12-16 11:40:07 101-1: delete previous replication snapshot '__replicate_101-1_1639649401__' on local-zfs:subvol-101-disk-0
2021-12-16 11:40:07 101-1: (remote_finalize_local_job) delete stale replication snapshot '__replicate_101-1_1639649401__' on local-zfs:subvol-101-disk-0
2021-12-16 11:40:08 101-1: end replication job

Backup: half an hour

INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2021-12-16 12:00:01
INFO: status = running
INFO: CT Name: xx.xx.com
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/101/2021-12-16T11:00:01Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1647308_101/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp1647308_101/etc/vzdump/pct.fw root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1639652401 --repository root@pam@xx.xx.xx.xx:store1
INFO: Starting backup: ct/101/2021-12-16T11:00:01Z
INFO: Client name: px1
INFO: Starting backup protocol: Thu Dec 16 12:00:06 2021
INFO: Downloading previous manifest (Thu Dec 16 06:00:01 2021)
INFO: Upload config file '/var/tmp/vzdumptmp1647308_101/etc/vzdump/pct.conf' to 'root@pam@xx.xx.xx.xx:8007:store1' as pct.conf.blob
INFO: Upload config file '/var/tmp/vzdumptmp1647308_101/etc/vzdump/pct.fw' to 'root@pam@xx.xx.xx.xx:8007:store1' as fw.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@xx.xx.xx.xx:8007:store1' as root.pxar.didx
INFO: root.pxar: had to backup 922.843 MiB of 40.359 GiB (compressed 338.05 MiB) in 1783.17s
INFO: root.pxar: average backup speed: 529.948 KiB/s
INFO: root.pxar: backup was done incrementally, reused 39.458 GiB (97.8%)
INFO: Uploaded backup catalog (21.545 MiB)
INFO: Duration: 1783.42s
INFO: End Time: Thu Dec 16 12:29:49 2021
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 101 (00:29:51)
INFO: Backup finished at 2021-12-16 12:29:52

BW is limited to 100M, for replication and also for backup (not at night, but speed stay similar).

There must be something weird on my setup

 

[Dunuin]

Other than a VM a LXC can't use dirty bitmapping so each time the complete LXC needs to be read and hashed. Only after that PBS can calculate what has changed and send that changed data incrementally. So if you got a 100GB LXC and just 1MB has changed, PBS still needs to read/hash the full 100GB each time. So I guess the actual incremental transfer is fast but reading in the complete LXC before that is what takes so long.

 

[liane]

Thank you for this explanation, but couldn't PBS use the same mechanics as replication for backup?

 

[Dunuin]

No, because PBS is using deduplication. It needs to save all data as deduplicated chunks.

 

[liane]

Well, I had to ask, the good news is that my setup is ok ;-)

Thank you