Slow dump on LXC

[Andrii]

Migrated all OpenVZ to LXC and can't use it because can't create a backups.
No snapshots? OK. We try to use suspend. But 1GB dump creating 5-10 hours on NFS. Tested on the same node KVM and created a dump during several minutes.

How to use this LXC?

KVM - 30 seconds and 12% was done. Why it can't be on LXC like this?

There is one solution. Need to create a local tmpdir. The dump will be fast, but if you try to backup 10GB example the IO will be 99% when this tmpdir will be erasing.

 

[manu]

Maybe you should create the tmpdir on a separate disk, so te tmpdir deleting does not block all pending IOs to your PVE host ?

 

[Andrii]

It is possible. But it will be an additional cost. Drive for tmpdir only…

 

[Andrii]

It should work correctly to create a backup in suspend mode. But it was not.

 

[Ovidiu]

I have the exact same problem, why are LXC backups taking 10 times longer than KVM backups? Any ideas?

Look at the 2 KVMs I marked, 103 is a Win10 machine and 111 is a Centos VM. They are so fast. All the rest are LXC.

 

[manu]

for this info you should have a look at the log of the backup task
usually it boils down to disks where the read are slows

 

[Ovidiu]

for this info you should have a look at the log of the backup task
usually it boils down to disks where the read are slows

Thanks but not sure how that helps. Let me add that this is a hardware raid, a simple LSI card but with BBU and 512MB cache and the backups and the disks all reside on the same 2 mirrored hard drives.

The point is, how can reads only be slow for LXC but not for KVM?

 

[dcsapak]

if you backup your containers to nfs, please create a "tmpdir" directory entry in /etc/vzdump.conf (would be best if this is on local storage)
else it creates a tmpdir on the nfs, copies everything there, and when creating the backup archive it gets read and written to the nfs server again...

 

[Ovidiu]

Thanks, that is very helpful in case I try to backup to NFS again but its not the case. like I said, I backup to "local".

btw. the backup speed "is slow" right? Its not supposed to take that long for LXC, correct?

 

[dcsapak]

The point is, how can reads only be slow for LXC but not for KVM?

well this can happen, as with kvm, the disk gets copied at a block level and for lxc this happens at the file level (so many small files can slow things down)

Thanks, that is very helpful in case I try to backup to NFS again but its not the case. like I said, I backup to "local".

ok i missed it in one of your last posts ( the first post says nfs)

btw. the backup speed "is slow" right? Its not supposed to take that long for LXC, correct?

this depends on the container, storage, hw, etc.

what kind of disks do you use? what kind of filesystem ? (default? ext4? barriers enabled? )
are your container heavy on io?

 

[Ovidiu]

Sorry, you are right, the original poster is backing up to NFS, I joined his thread because he also experienced a different speed between LXC/KVM backups.

with kvm, the disk gets copied at a block level and for lxc this happens at the file level

thanks, I think this explains everything :-(

this depends on the container, storage, hw, etc.

what kind of disks do you use? what kind of filesystem ? (default? ext4? barriers enabled? )
are your container heavy on io?

Its an entry-level server, with 2 mirrored SATA 7200RPM consumer HDs on a LSI raid controller with a BBU and caching enabled.
Proxmox is installed with the defaults so ext3 and as far as I know barriers were disabled by default. The container taking so long to backup ID 100 is serving a few websites but has very low usage and also an email server but with no more than 200 emails per day so its basically idling along except for when the backup occurs.

I am seriously considering switching over to a KVM as this backup runs 9 hours and slows everything to a halt and triggers all sorts of warnings to be sent out to me via monitoring tools.

 

[dcsapak]

Proxmox is installed with the defaults so ext3 and as far as I know barriers were disabled by default.

proxmox >= 4.x uses ext4 by default ( which enables barriers by default) so since you have a raid card with bbu, you should try to disable barriers.

 

[Ovidiu]

proxmox >= 4.x uses ext4 by default ( which enables barriers by default) so since you have a raid card with bbu, you should try to disable barriers.

Thanks, I realized my reply was incomplete, this was upgraded from an older 3.x Proxmox version as far as I can remember, its definitely ext3.
from my fstab:

cat /etc/fstab
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/sda2       /       ext3    errors=remount-ro       0       1
/dev/sda3       swap    swap    defaults        0       0
/dev/pve/data   /var/lib/vz     ext3    defaults        1       2
proc            /proc   proc    defaults        0       0
sysfs           /sys    sysfs   defaults        0       0