Container slow backup

[ricardoj]

Hi,

Here in my lab I noticed that container backup are very slow compared to VM backups.

I mean even a small container takes much more time to backup them a "bigger" VM.

Tests were made on version 5.3 and local ZFS storage

Can someone comment on that ?

Regards,

Ricardo Jorge

 

[mira]

Please post the container config ('pct config <ctid>') and the log of a slow backup task.

 

[ricardoj]

Hi Mira,

root@pve-01:~# pct config 100
arch: amd64
cmode: tty
cores: 3
hostname: PATRU01
lock: backup
memory: 3072
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=62:2D:1B:37:EB:BC,ip=192.168.0.211/24,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-100-disk-0,size=10G
swap: 1024
root@pve-01:~#

This is part of a backup job

INFO: Starting Backup of VM 100 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: PATRU01
INFO: creating archive '/mnt/pve/NFS_Novo/dump/vzdump-lxc-100-2019_04_13-19_00_02.tar.lzo'
INFO: Total bytes written: 7406059520 (6.9GiB, 621KiB/s)
INFO: archive file size: 5.21GB
INFO: Finished Backup of VM 100 (03:14:22)

As you can see for a 10 GBytes container backup tooks 3 hours !

Now for a 20 GBytes VM, it tooks 09 minutes

VM backup

INFO: starting new backup job: vzdump 200 --storage local --compress lzo --remove 0 --mode stop --node pve-01
INFO: Starting Backup of VM 200 (qemu)
INFO: status = stopped
INFO: update VM 200: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: WS-1804
INFO: include disk 'scsi0' 'local-zfs:vm-200-disk-1' 20G
INFO: skip unused drive 'NFS_Novo:200/vm-200-disk-0.qcow2' (not included into backup)
INFO: skip unused drive 'local-zfs:vm-200-disk-0' (not included into backup)
INFO: skip unused drive 'NFS_Novo:200/vm-200-disk-0.raw' (not included into backup)
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-200-2019_04_13-11_43_31.vma.lzo'
INFO: starting kvm to execute backup task
Use of uninitialized value $type in string eq at /usr/share/perl5/PVE/QemuServer.pm line 2026.
INFO: started backup task '12ae14db-1f29-4475-8d5b-1258a3ac505f'

...

INFO: transferred 21474 MB in 547 seconds (39 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 13.15GB
INFO: Finished Backup of VM 200 (00:09:10)
INFO: Backup job finished successfully
TASK OK

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Those backups were made on PVE 5.3 with ZFS

Today I did some tests with PVE 5.4 with BtrFS and the same container backup took only 1 minute and 45 seconds

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
INFO: starting new backup job: vzdump 1000 --storage NFS_PVE --compress lzo --node pve-01 --remove
0 --mode stop
INFO: Starting Backup of VM 1000 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: PATRU01
INFO: creating archive '/mnt/pve/NFS_PVE/dump/vzdump-lxc-1000-2019_04_18-09_36_32.tar.lzo'
INFO: Total bytes written: 7409602560 (7.0GiB, 78MiB/s)
INFO: archive file size: 5.21GB
INFO: Finished Backup of VM 1000 (00:01:45)
INFO: Backup job finished successfully
TASK OK

Target backup is always the same NFS hardware which is running OMV 4.1.22 with BtrFS as storage ( RAID 1 ).

If here is no code change from PVE 5.3 to PVE 5.4 for container backup, the only difference at my side is the PVE filesystem.

PVE 5.3 is running ZFS and PVE 5.4 is running BtrFS as local storage.

Regards,

Ricardo Jorge

 

[ricardoj]

Hi,

Info about NFS

root@pve-01:~# nfsstat -m
/mnt/pve/NFS_Novo from 192.168.0.216:/NFS_PVE
Flags: rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.108,local_lock=none,addr=192.168.0.216

root@pve-01:~#

Regards,

Ricardo Jorge

 

[ricardoj]

Hi,

I just did a new backup ( same container ) :

root@pve-01:~# pct config 100
arch: amd64
cmode: tty
cores: 3
hostname: PATRU01
memory: 3072
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=62:2D:1B:37:EB:BC,ip=192.168.0.211/24,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-100-disk-0,size=10G
swap: 1024
root@pve-01:~#

root@pve-01:~# pveperf
CPU BOGOMIPS: 42190.02
REGEX/SECOND: 1354425
HD SIZE: 677.92 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 66.53
DNS EXT: 126.96 ms
DNS INT: 61.61 ms (local)
root@pve-01:~#

root@pve-01:~# pveversion
pve-manager/5.3-5/97ae681d (running kernel: 4.15.18-9-pve)
root@pve-01:~#

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
INFO: starting new backup job: vzdump 100 --storage NFS_Novo --compress lzo --remove 0 --node pve-01 --mode stop
INFO: Starting Backup of VM 100 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: PATRU01
INFO: creating archive '/mnt/pve/NFS_Novo/dump/vzdump-lxc-100-2019_04_18-10_37_27.tar.lzo'
INFO: Total bytes written: 7406059520 (6.9GiB, 645KiB/s)
INFO: archive file size: 5.21GB
INFO: Finished Backup of VM 100 (03:07:06)
INFO: Backup job finished successfully
TASK OK
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Regards,

Ricardo Jorge

 

[mira]

Please post the complete output of 'pveversion -v', your storage config (/etc/pve/storage.cfg), the vzdump config (/etc/vzdump.conf) and some information regarding your hardware. (CPU, RAM, Disks, NICs, and more if possible)

 

[ricardoj]

Hi Mira,

I'm sending the information you asked me.

I know this machine is far from the best hardware but keep in in mind that :

a) - This same machine can backup a VM with 20 GBytes in less than 10 minutes but takes at least 3 hours to backup a container with 10 GBytes

b) - This same machine running PVE 5.4 can backup the exact same container in less than 2 minutes if I have BtrFS installed on it.

Question : Is there any difference for container backup between PVE 5.3 and PVE 5.4 ?

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

This machine has 16 GBytes of RAM

==== general system info ====

# hostname
pve-01

# pveversion --verbose
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.0.108 pve-01.local pve-01

# top -b -n 1 | head -n 15
top - 15:29:55 up 5:13, 1 user, load average: 0.14, 0.11, 0.03
Tasks: 225 total, 1 running, 161 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 3.2 sy, 0.0 ni, 95.7 id, 0.6 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16376164 total, 14966100 free, 1268580 used, 141484 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 14840344 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23608 root 20 0 43264 3128 2516 R 6.2 0.0 0:00.01 top
1 root 20 0 57392 5608 3796 S 0.0 0.0 0:02.59 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:02.45 kthreadd
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_+
7 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/0
8 root 20 0 0 0 0 I 0.0 0.0 0:02.78 rcu_sched
9 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 6
On-line CPU(s) list: 0-5
Thread(s) per core: 2
Core(s) per socket: 3
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 21
Model: 2
Model name: AMD FX(tm)-6300 Six-Core Processor
Stepping: 0
CPU MHz: 1823.652
CPU max MHz: 3500.0000
CPU min MHz: 1400.0000
BogoMIPS: 7031.67
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 8192K
NUMA node0 CPU(s): 0-5
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold


==== info about storage ====

# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content backup,iso,vztmpl
maxfiles 0
shared 0

zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1

nfs: NFS_Novo
export /NFS_PVE
path /mnt/pve/NFS_Novo
server 192.168.0.216
content backup,rootdir,vztmpl,iso,images
maxfiles 5
options vers=4

# pvesm status
Name Type Status Total Used Available %
NFS_Novo nfs active 3907017728 1110876672 2794676224 28.43%
local dir active 710847744 473404672 237443072 66.60%
local-zfs zfspool active 469227484 231784328 237443156 49.40%

==== info about bios ====

# dmidecode -t bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.6 present.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: American Megatrends Inc.
Version: P1.20
Release Date: 06/04/2012
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 1024 kB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
BIOS Revision: 8.15


==== info about disks ====

# lsblk --ascii
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
|-sda1 8:1 0 1007K 0 part
|-sda2 8:2 0 512M 0 part
`-sda3 8:3 0 931G 0 part
sdb 8:16 0 931.5G 0 disk
|-sdb1 8:17 0 1007K 0 part
|-sdb2 8:18 0 512M 0 part
`-sdb3 8:19 0 931G 0 part
sr0 11:0 1 1024M 0 rom
zd0 230:0 0 8G 0 disk
|-zd0p1 230:1 0 6G 0 part
|-zd0p2 230:2 0 1K 0 part
`-zd0p5 230:5 0 2G 0 part
zd16 230:16 0 20G 0 disk
`-zd16p1 230:17 0 20G 0 part
zd32 230:32 0 160G 0 disk
`-zd32p1 230:33 0 160G 0 part
zd48 230:48 0 32G 0 disk
`-zd48p1 230:49 0 32G 0 part

 

[mira]

Please post the vzdump config as well (/etc/vzdump.conf). Try creating the backup on a local storage instead of the NFS to see if that changes anything.

 

[ricardoj]

Hi Mira,

Is there any difference for container backup between PVE 5.3 and PVE 5.4 ?

Regards,

Ricardo Jorge

 

[ricardoj]

Hi Mira,

Here are the informations you asked me.

As you can see the time difference with and without NAS ( network ) is minimum.

The only really big difference I see for container backup was changing the filesystem from ZFS to BtrFS.

PVE 5.3 is running ZFS and PVE 5.4 is running BtrFS.

That is why I'm asking you if there is any difference in container backup from PVE versions.

For VMs there is no noticeable time difference for backup with ZFS and BtrFS.

I can see that VMs are running smoothly under BtrFS and I also have better VM performance.

Regards,

Ricardo Jorge

====================================

INFO: starting new backup job: vzdump 100 --remove 0 --mode stop --storage local --node pve-01 --compress lzo
INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp3951 for temporary files
INFO: Starting Backup of VM 100 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: PATRU01
INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2019_03_24-09_29_53.tar.lzo'
INFO: Total bytes written: 7406059520 (6.9GiB, 637KiB/s)
INFO: archive file size: 5.21GB
INFO: Finished Backup of VM 100 (03:09:30)
INFO: Backup job finished successfully
TASK OK

====================================

root@pve-01:~# cat /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N:root@pve-01:~#

====================================

root@pve-01:/var/lib/vz/dump# cat vzdump-lxc-100-2019_03_24-09_29_53.log
2019-03-24 09:29:53 INFO: Starting Backup of VM 100 (lxc)
2019-03-24 09:29:53 INFO: status = stopped
2019-03-24 09:29:53 INFO: backup mode: stop
2019-03-24 09:29:53 INFO: ionice priority: 7
2019-03-24 09:29:53 INFO: CT Name: PATRU01
2019-03-24 09:29:53 INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2019_03_24-09_29_53.tar.lzo'
2019-03-24 12:39:23 INFO: Total bytes written: 7406059520 (6.9GiB, 637KiB/s)
2019-03-24 12:39:23 INFO: archive file size: 5.21GB
2019-03-24 12:39:23 INFO: Finished Backup of VM 100 (03:09:30)
root@pve-01:/var/lib/vz/dump#

 

[ricardoj]

Hi Mira,

I see you focusing too much on my lab hardware

The same ratio for container X VM backup time happen in a much more "powerful" hardware

It is a Ryzen 7 2700 with 64 GB RAM and faster SATA disks PVE 5.2 and ZFS

A container backup with 325 MBytes took 10 minutes

A VM backup with 6.91 GBytes took 5:48 minutes

It is not the hardware and you can simulate this in your lab.

With better hardware you have faster backup but not better ratio.

Regards,

Ricardo Jorge

=====================================

INFO: Starting Backup of VM 100 (lxc)
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: dns-01
INFO: stopping vm
INFO: creating archive '/mnt/pve/NFS-NAS01/dump/vzdump-lxc-100-2019_04_20-02_45_02.tar.lzo'
INFO: Total bytes written: 691015680 (660MiB, 1.1MiB/s)
INFO: archive file size: 325MB
INFO: delete old backup '/mnt/pve/NFS-NAS01/dump/vzdump-lxc-100-2019_03_16-02_45_01.tar.lzo'
INFO: restarting vm
INFO: vm is online again after 634 seconds
INFO: Finished Backup of VM 100 (00:10:34)

===================================

INFO: Starting Backup of VM 105 (qemu)
INFO: status = stopped
INFO: update VM 105: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Ubuntu-01
INFO: include disk 'virtio0' 'local-zfs:vm-105-disk-1' 12G
INFO: creating archive '/mnt/pve/NFS-NAS01/dump/vzdump-qemu-105-2019_04_20-03_17_21.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task '0e475ef4-2c73-4109-9cb1-581218a6b253'
INFO: status: 1% (150011904/12884901888), sparse 0% (29650944), duration 3, read/write 50/40 MB/s
INFO: status: 2% (361693184/12884901888), sparse 0% (30461952), duration 6, read/write 70/70 MB/s
...
INFO: status: 98% (12738166784/12884901888), sparse 2% (364621824), duration 336, read/write 49/47 MB/s
INFO: status: 100% (12884901888/12884901888), sparse 2% (367149056), duration 339, read/write 48/48 MB/s
INFO: transferred 12884 MB in 339 seconds (38 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 6.91GB
INFO: delete old backup '/mnt/pve/NFS-NAS01/dump/vzdump-qemu-105-2019_03_16-03_28_57.vma.lzo'
INFO: Finished Backup of VM 105 (00:05:48)