Backup 3x times bigger than VM content

[keyjey]

Hi all,

There's an issue that I can't really understand.

We have one KVM VM (W2k3) with a disk of 32G in LVM. The used disk inside the VM is about 9G, and the backup make 23'5G.Who can understand this ?

We tried to move it to a /directory storage converting it to qcow2, to defrag the entire disk ...and always the same result: 23'5G.

[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 312 --remove 0 --mode snapshot --compress lzo --storage backups_TMP --node hw01[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 312 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = stopped[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: update VM 312: -lock backup[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: stop[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/backups/dump/vzdump-qemu-312-2013_12_19-00_11_29.vma.lzo'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: starting kvm to execute backup task[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: started backup task 'e5146c78-b255-49cc-b4b7-11f26f2dfa6f'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 2% (830603264/34359738368), sparse 0% (59838464), duration 3, 276/256 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 4% (1620901888/34359738368), sparse 0% (68513792), duration 6, 263/260 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]...
INFO: status: 96% (33205846016/34359738368), sparse 10% (3646115840), duration 129, 223/223 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 98% (34003812352/34359738368), sparse 10% (3646181376), duration 132, 265/265 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (34359738368/34359738368), sparse 10% (3658600448), duration 134, 177/171 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: transferred 34359 MB in 134 seconds (256 MB/s)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: stopping kvm after backup task[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: archive file size: 23.52GB[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Finished Backup of VM 312 (00:02:17)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Backup job finished successfully[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK[/FONT][/COLOR]

And after trying almost everything, we tried to make a backup using gzip instead of LZO and almost the same, but this time backup needed 7x more time for a similar result:

[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 312 --remove 0 --mode snapshot --compress gzip --storage backups_TMP --node hw01[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 312 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: update VM 312: -lock backup[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/backups/dump/vzdump-qemu-312-2013_12_19-00_50_59.vma.gz'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: started backup task '38f5e637-c14d-44bc-a9c9-0f99bc8e4271'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 0% (106430464/34359738368), sparse 0% (446464), duration 3, 35/35 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 1% (368705536/34359738368), sparse 0% (55898112), duration 10, 37/29 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 2% (710803456/34359738368), sparse 0% (59432960), duration 22, 28/28 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 3% (1053229056/34359738368), sparse 0% (62881792), duration 34, 28/28 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]...
INFO: status: 98% (33673248768/34359738368), sparse 10% (3645939712), duration 1047, 29/29 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 99% (34046345216/34359738368), sparse 10% (3646005248), duration 1059, 31/31 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (34359738368/34359738368), sparse 10% (3658424320), duration 1071, 26/25 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (34359738368/34359738368), sparse 10% (3658424320), duration 1072, 0/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: transferred 34359 MB in 1072 seconds (32 MB/s)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: archive file size: 22.03GB[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Finished Backup of VM 312 (00:17:52)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Backup job finished successfully[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK[/FONT][/COLOR]

We have a similar VM (same template) with same disk, OS, etc and the 14GBs used space makes a 9GBs backup. This means there's something weird with that VM disk.

Any ideas ? maybe this is normal ? A VM with 9GBs of total data generating backups of 24GBs ?

Thanks

 

[alek]

vzdump archives all blocks with any data, even deleted.
You need to clean not used blocks with zeros.
I use sysinternals "sdelete -c" for this on windows machines.
For Linux I use "dd if=/dev/zero of=.junk; sync; rm -f .junk"
Also I use separate disk with nobackup flag for swap - saves additional backup space and time.

 

[keyjey]

I didn't think in that .... after cleaning the empty space this is the result:

INFO: transferred 34359 MB in 145 seconds (236 MB/s)
INFO: archive file size: 5.05GB

Thanks a lot !

 

[Slamdunk]

For the future reader, there is also the official guide on how to shrink backups:

https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files

 

[VladVons]

Thank a LOT!