Backup CT slower then VM

[Kaboom]

Hi All,

I did backup all my Containers and VM's (encrypted), now I am running a second round incrementally. Backing up Vm's are lightning fast, but the Cointainer backups are really slow compared to the VM's. Is this normal ?

Thanks

 

[vesalius]

LXC/Container backup slowdown is a known problem for PBS. No fix is planned as of yet.

https://forum.proxmox.com/threads/container-backup-is-slow.77179/
https://forum.proxmox.com/threads/lxc-containers-backing-up-incredibly-slow.79188/
https://forum.proxmox.com/threads/no-differantial-container-backup-with-big-containers.75676/

Hoping Proxmox can figure something out eventually.

 

[Kaboom]

Thanks for the update Vesalius, hopefully there will be a solution soon.

 

[fabian]

it's still incremental, it's just that if your read speed is low (or you have huge amounts of data) it will still take a while. VMs simply have the advantage that there is a layer of abstraction between the guest and the storage that can monitor for us which parts of the disks have changed, so we can skip the 'read everything to see what has changed' part.

 

[carsten2]

it's still incremental, it's just that if your read speed is low (or you have huge amounts of data) it will still take a while. VMs simply have the advantage that there is a layer of abstraction between the guest and the storage that can monitor for us which parts of the disks have changed, so we can skip the 'read everything to see what has changed' part.

Actually it is not the problem, that differences are not known, but it seems to be a design deficiency in the container backup format, which is quite different from the VM backup format.

 

[fabian]

Actually it is not the problem, that differences are not known, but it seems to be a design deficiency in the container backup format, which is quite different from the VM backup format.

yes and no. the problem is two-fold:
- container/host backups work on a file level, not block level, so we need to convert 1..N files to a single chunk (we do this by generating an archive stream, which then gets chunked by a rolling hash)
- for containers/directory hierarchies, there is no sane storage-agnostic way to cheaply and consistently track the differences (even ZFS snapshots don't count, since they cost storage space and are thus not cheap if the delta gets big!)

since there is no solution/implementation for the second problem, it does not make sense to implement any possible, complicated solution to solve the 're-use most of previous pxar stream and only replace chunks containing files A, B and D' problem. I'm not saying that it will never happen, but this is not as trivial as it sounds

 

[frkb]

I have noticed that Debian/Ubuntu container backups two time faster as Fedora CT.
To be sure, I have reinstall a Fedora CT with Ubuntu.
System install, software install, copy data.

Type	Distro	Size	Duration
First Backup	Fedora	27.02GB	00:45:23
First Backup	Ubuntu	23.67GB	00:42:29
Incremental	Fedora	27.02GB	00:05:14
Incremental	Ubuntu	23.67GB	00:02:23

On the other hand it is cool to be able to retrieve just one file directly in a CT backup. Which is impossible in a VM backup.
Please let this as it is.

 

[fabian]

I have noticed that Debian/Ubuntu container backups two time faster as Fedora CT.
To be sure, I have reinstall a Fedora CT with Ubuntu.
System install, software install, copy data.

Type	Distro	Size	Duration
First Backup	Fedora	27.02GB	00:45:23
First Backup	Ubuntu	23.67GB	00:42:29
Incremental	Fedora	27.02GB	00:05:14
Incremental	Ubuntu	23.67GB	00:02:23

On the other hand it is cool to be able to retrieve just one file directly in a CT backup. Which is impossible in a VM backup.
Please let this as it is.

that's probably an artifact of how the (default) container templates are setup - we don't do anything special with different container os types... the backup log should contain more detailed info about how much data changed since the last backup.

 

[frkb]

If you can see some thing :

Ubuntu :

INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2020-12-19 12:31:04
INFO: status = running
INFO: CT Name: lcdb2
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: PYHOOK-PRESTOP: vm=101, Type=lxc status=FROZEN
INFO: create storage snapshot 'vzdump'
INFO: PYHOOK-POSTSTOP: vm=101, Type=lxc, status=UNFROZEN
INFO: PYHOOK-POSTSTOP: vm=101, Type=lxc status=UNFROZEN temps d'arret=1.28 sec.
INFO: creating Proxmox Backup Server archive 'ct/101/2020-12-19T11:31:04Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2581_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1608377464 --repository backup@pbs@tobozon.frkb.fr:ds1
INFO: Starting backup: ct/101/2020-12-19T11:31:04Z
INFO: Client name: azimuth2
INFO: Starting backup protocol: Sat Dec 19 12:31:06 2020
INFO: Downloading previous manifest (Sat Dec 19 11:33:54 2020)
INFO: Upload config file '/var/tmp/vzdumptmp2581_101/etc/vzdump/pct.conf' to 'backup@pbs@tobozon.frkb.fr:8007:ds1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@tobozon.frkb.fr:8007:ds1' as root.pxar.didx
INFO: root.pxar: had to upload 34.70 MiB of 23.67 GiB in 137.88s, average speed 257.72 KiB/s).
INFO: root.pxar: backup was done incrementally, reused 23.63 GiB (99.9%)
INFO: Uploaded backup catalog (1.81 MiB)
INFO: Duration: 140.37s
INFO: End Time: Sat Dec 19 12:33:26 2020
INFO: PYHOOK-POSTSTOP: vm=101, Type=lxc status=UNFROZEN temps d'arret=1608377605.97 sec.
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 101 (00:02:23)
INFO: Backup finished at 2020-12-19 12:33:27

Fedora :

INFO: Starting Backup of VM 121 (lxc)
INFO: Backup started at 2020-12-19 13:38:42
INFO: status = running
INFO: CT Name: lcdb.eu
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: PYHOOK-MYSQL: Base mysql trouv?e ==> LOCK
INFO: PYHOOK-PRESTOP: vm=121, Type=lxc status=FROZEN
INFO: create storage snapshot 'vzdump'
INFO: PYHOOK-POSTSTOP: vm=121, Type=lxc, status=UNFROZEN
INFO: PYHOOK-MYSQL: Base mysql trouv?e ==> UNLOCK
INFO: PYHOOK-POSTSTOP: vm=121, Type=lxc status=UNFROZEN temps d'arret=1.68 sec.
INFO: creating Proxmox Backup Server archive 'ct/121/2020-12-19T12:38:42Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2581_121/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 121 --backup-time 1608381522 --repository backup@pbs@tobozon.frkb.fr:ds1
INFO: Starting backup: ct/121/2020-12-19T12:38:42Z
INFO: Client name: azimuth2
INFO: Starting backup protocol: Sat Dec 19 13:38:44 2020
INFO: Downloading previous manifest (Sat Dec 19 02:09:42 2020)
INFO: Upload config file '/var/tmp/vzdumptmp2581_121/etc/vzdump/pct.conf' to 'backup@pbs@tobozon.frkb.fr:8007:ds1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@tobozon.frkb.fr:8007:ds1' as root.pxar.didx
INFO: root.pxar: had to upload 277.63 MiB of 27.36 GiB in 324.49s, average speed 876.13 KiB/s).
INFO: root.pxar: backup was done incrementally, reused 27.09 GiB (99.0%)
INFO: Uploaded backup catalog (3.13 MiB)
INFO: Duration: 327.64s
INFO: End Time: Sat Dec 19 13:44:12 2020
INFO: PYHOOK-POSTSTOP: vm=121, Type=lxc status=UNFROZEN temps d'arret=1608381851.5 sec.
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 121 (00:05:31)
INFO: Backup finished at 2020-12-19 13:44:13

I am about to testing backup from inside a container with backup client. I do this on the host and it finish in light speed. The only drawback could be the restoration.

 

[fabian]

yeah, you can clearly see the difference: 34.7 MB in one case vs 288.63MB in the other, and the total read+digested data is also higher for the second container.

 

[frkb]

Yes you are right. This containers are hosting Nextcloud. The Fedora is the living one while the Ubuntu is just a dead copy.
Meanwhile I have do comparison between vzdump and a backup from the inside of the container with proxmox-backup-client.
Duration is nearly the same.