LXC Containers Backing Up Incredibly Slow

[infinityM]

Hey Guys,

We have a LXC container which is about 2TB large, When backing up this server though it takes about 7+ hours to backup.
While I have several VM's which only take about 20 minutes to backup but is 5-6TB large...

Why are LXC's so slow to backup?

 

[matrix]

to help better, please provide your storage config cat /etc/pve/storage.cfg and your container config

 

[infinityM]

to help better, please provide your storage config cat /etc/pve/storage.cfg and your container config

Hey Matrix,

root@c6:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

rbd: Default
content images,rootdir
krbd 0
pool Default

nfs: NAS
export /volume1/Backups
path /mnt/pve/NAS
server 10.161.0.247
content snippets,vztmpl,backup,iso,rootdir,images
maxfiles 3

pbs: backups
datastore backups
server pmb.local
content backup
fingerprint # I removed the fingerprint
maxfiles 99
username root@pam

LXC Config

root@c6:~# cat /etc/pve/lxc/127.conf
arch: amd64
cores: 6
hostname: VPS127
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,gw=156.38.175.34,hwaddr=22:72:12:00:8F:59,ip=156.38.175.58/27,type=veth
onboot: 1
ostype: ubuntu
rootfs: Default:vm-127-disk-0,size=2000G
swap: 8192
unprivileged: 1

 

[matrix]

rbd: Default
content images,rootdir
krbd 0
pool Default

Other VMs are in the same storage?

 

[infinityM]

Other VMs are in the same storage?

Yes, all VM's and LXC's are on the Ceph Storage

 

[fabian]

because for containers, PBS needs to read and chunk all the data to find out what to upload, for (running) VMs there is a shortcut because we can tell Qemu to keep track of which chunks changed.

 

[infinityM]

because for containers, PBS needs to read and chunk all the data to find out what to upload, for (running) VMs there is a shortcut because we can tell Qemu to keep track of which chunks changed.

Hey Fabian,

Is there any plans to get around this problem?

 

[fabian]

there is no cheap solution for this - the short term would definitely be to move such workloads to VMs where there is this intermediate layer that can keep track of writes from the guest and speedup the backup.

 

[infinityM]

I assume you mean to convert the LXC to a QEMU VM?

 

[fabian]

yes, basically. then you only need to re-read all of the data when
- the VM was down (live-migrating to another node will now transfer the bitmap as well!)
- the encryption key or mode was changed

 

[fabian]

of course, increasing your read/chunk performance might also be an option depending on your current hardware

 

[infinityM]

of course, increasing your read/chunk performance might also be an option depending on your current hardware

How can one check/change this?

 

[fabian]

read: I/O benchmark (more/faster disks, to some degree more RAM)
chunk/hash: proxmox-backup-client benchmark (faster CPU/more cores)

 

[carsten2]

I opened a feature request https://bugzilla.proxmox.com/show_bug.cgi?id=3138 to speed up container backup, but it gets closed multiple times without discussion and even with wrong claims. The feature request is perfectly valid, because PBS would be the ONLY backup solution in the world, which needs several hours to backup UNCHANGED file systems. I claim that a dramatic improvement IS possible and it is even not too complicated to implement.

 

[tom]

@carsten2 No double posts please!