Increase backup speed for CT

[mir]

Hi all,

Just a little new years hint to how you are able to increase backup speed of CT's when backups are stored remote and especially when remote is NFS mounted.

1) in /etc/vzdump.conf on each node add: tmpdir: /var/lib/vz/vztmp

Above means the copy and assembly of the CT backup will be done local instead of remote.

With default settings:

vzdump 101 --remove 0 --mode snapshot --compress lzo --storage qnap_nfs --node esx2
INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --node esx2 --storage qnap_nfs
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: mode failure - unable to detect lvm volume group
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: bandwidth limit: 50000 KB/s
INFO: ionice priority: 7
INFO: starting first sync /mnt/pve/omnios_ib_nfs/private/101/ to /mnt/pve/qnap_nfs/dump/vzdump-openvz-101-2015_01_04-16_09_15.tmp
INFO: Number of files: 31066
INFO: Number of files transferred: 24306
INFO: Total file size: 1981306183 bytes
INFO: Total transferred file size: 1979086112 bytes
INFO: Literal data: 1979086112 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 731434
INFO: File list generation time: 0.002 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 1981110671
INFO: Total bytes received: 484812
INFO: sent 1981110671 bytes  received 484812 bytes  15301895.62 bytes/sec
INFO: total size is 1981306183  speedup is 1.00
INFO: first sync finished (129 seconds)
INFO: suspend vm
INFO: Setting up checkpoint...
INFO:     suspend...
INFO:     get context...
INFO: Checkpointing completed successfully
INFO: starting final sync /mnt/pve/omnios_ib_nfs/private/101/ to /mnt/pve/qnap_nfs/dump/vzdump-openvz-101-2015_01_04-16_09_15.tmp
INFO: Number of files: 31066
INFO: Number of files transferred: 0
INFO: Total file size: 1981306183 bytes
INFO: Total transferred file size: 0 bytes
INFO: Literal data: 0 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 731434
INFO: File list generation time: 0.004 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 733859
INFO: Total bytes received: 2424
INFO: sent 733859 bytes  received 2424 bytes  133869.64 bytes/sec
INFO: total size is 1981306183  speedup is 2690.96
INFO: final sync finished (5 seconds)
INFO: resume vm
INFO: Resuming...
INFO: vm is online again after 5 seconds
INFO: creating archive '/mnt/pve/qnap_nfs/dump/vzdump-openvz-101-2015_01_04-16_09_15.tar.lzo'
INFO: Total bytes written: 2001203200 (1.9GiB, 29MiB/s)
INFO: archive file size: 572MB
INFO: Finished Backup of VM 101 (00:03:50)
INFO: Backup job finished successfully

With my improved setting:

vzdump 101 --remove 0 --mode snapshot --compress lzo --storage qnap_nfs --node esx2
INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --node esx2 --storage qnap_nfs
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: mode failure - unable to detect lvm volume group
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: bandwidth limit: 50000 KB/s
INFO: ionice priority: 7
INFO: starting first sync /mnt/pve/omnios_ib_nfs/private/101/ to /var/lib/vz/vztmp/vzdumptmp684133
INFO: Number of files: 31066
INFO: Number of files transferred: 24306
INFO: Total file size: 1981310141 bytes
INFO: Total transferred file size: 1979090070 bytes
INFO: Literal data: 1979090070 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 731434
INFO: File list generation time: 0.003 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 1981114629
INFO: Total bytes received: 484812
INFO: sent 1981114629 bytes  received 484812 bytes  44530324.52 bytes/sec
INFO: total size is 1981310141  speedup is 1.00
INFO: first sync finished (44 seconds)
INFO: suspend vm
INFO: Setting up checkpoint...
INFO:     suspend...
INFO:     get context...
INFO: Checkpointing completed successfully
INFO: starting final sync /mnt/pve/omnios_ib_nfs/private/101/ to /var/lib/vz/vztmp/vzdumptmp684133
INFO: Number of files: 31066
INFO: Number of files transferred: 0
INFO: Total file size: 1981310141 bytes
INFO: Total transferred file size: 0 bytes
INFO: Literal data: 0 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 731434
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 733859
INFO: Total bytes received: 2424
INFO: sent 733859 bytes  received 2424 bytes  1472566.00 bytes/sec
INFO: total size is 1981310141  speedup is 2690.96
INFO: final sync finished (0 seconds)
INFO: resume vm
INFO: Resuming...
INFO: vm is online again after 0 seconds
INFO: creating archive '/mnt/pve/qnap_nfs/dump/vzdump-openvz-101-2015_01_04-16_13_55.tar.lzo'
INFO: Total bytes written: 2001213440 (1.9GiB, 46MiB/s)
INFO: archive file size: 572MB
INFO: Finished Backup of VM 101 (00:01:35)
INFO: Backup job finished successfully

Improved duration from 00:03:50 to 00:01:35. Almost 60% faster.
Improved bandwidth from 29MiB/s to 46MiB/s. Almost 45% more bandwidth.
Improved time when CT is suspended from 5 sec to 0 sec.

NFS is running on a fairly slow Qnap with RAID 1 on SATA3 green WD 1.5 TB over a single gigabit line.


Happy new year to all

 

[awesome]

It's works. I have got around 50% speed increase. Thank you.

 

[m.ardito]

1) in /etc/vzdump.conf on each node add: tmpdir: /var/lib/vz/vztmp
Above means the copy and assembly of the CT backup will be done local instead of remote.
...
Happy new year to all

nice catch Mir! Thanks!

I would just stress out that doing this, people should make sure to have enough local disk space in /var/lib/vz/vztmp, if I got it well... or in low disk space conditions, would it revert to usual direct-to-storage slower backup?

Marco

 

[mir]

would it revert to usual direct-to-storage slower backup?
Marco

No, if disk space is not sufficient the backup will stop with an error notice.