Backup/Restore Issue with PVE2.2

rengiared

Active Member
Sep 8, 2010
96
0
26
Austria
This week i installed a new PVE Host in a new environment with PVE 2.2 ISO and all updates and i found an annoying behaviour in the backup/restore-procedure.

I created a new guest with 2 disks, one 400GiB and one 200GiB and even if i start a backup of the complete empty vm it takes about half an hour.

here some facts about the host and the guest
Code:
pveversion -v
pve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1
Code:
bootdisk: ide0
cores: 4
ide0: local:102/vm-102-disk-1.qcow2,size=400G
ide1: local:102/vm-102-disk-2.qcow2,size=200G
ide2: pve:iso/ATIH2010.iso,media=cdrom,size=78656K
memory: 24576
name: guest
net0: e1000=EE:22:17:C8:F3:BD,bridge=vmbr0
ostype: win7
sockets: 2
Code:
INFO: starting new backup job: vzdump 102 --remove 0 --mode snapshot --compress lzo --storage nem --node pvenem01
INFO: Starting Backup of VM 102 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nem/dump/vzdump-qemu-102-2013_02_20-14_43_13.tar.lzo'
INFO: adding '/mnt/pve/nem/dump/vzdump-qemu-102-2013_02_20-14_43_13.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/var/lib/vz/images/102/vm-102-disk-1.qcow2' to archive ('vm-disk-ide0.qcow2')
INFO: adding '/var/lib/vz/images/102/vm-102-disk-2.qcow2' to archive ('vm-disk-ide1.qcow2')
INFO: Total bytes written: 644343925760 (315.45 MiB/s)
INFO: archive file size: 2.44GB
INFO: Finished Backup of VM 102 (00:32:30)
INFO: Backup job finished successfully
TASK OK

it looks like the backup is going through the whole empty space and packing it up

if i do the same on a guest created with pve2.1, then upgraded to the latest pve2.2 version its much faster (in this case there are already about 30GiB data in the guest)
Code:
INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --storage nem --node pvenem01
INFO: Starting Backup of VM 101 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nem/dump/vzdump-qemu-101-2013_02_20-13_30_49.tar.lzo'
INFO: adding '/mnt/pve/nem/dump/vzdump-qemu-101-2013_02_20-13_30_49.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/var/lib/vz/images/101/vm-101-disk-1.qcow2' to archive ('vm-disk-ide0.qcow2')
INFO: adding '/var/lib/vz/images/101/vm-101-disk-2.qcow2' to archive ('vm-disk-ide1.qcow2')
INFO: Total bytes written: 32336382976 (110.53 MiB/s)
INFO: archive file size: 8.49GB
INFO: Finished Backup of VM 101 (00:05:11)
INFO: Backup job finished successfully
TASK OK

i know that the behaviour on creating a qcow between 2.1 and 2.2 changed, cause in 2.1 qcows grew continually and with 2.2 they take the place they need immediately

the problem now is that it takes 25minutes longer for "nothing".
of course, you could say make the hdd smaller and expand it if necessary, but in my opinion thats not really practically
the overall 600GiB will be filled with time and the problem with lost backup-time for empty space will shrink cause then its backuping real data, but there will be always some lost time that can multiply to hours (with more machines and the more space is assigned)

is this behaviour known or even wanted?

for the end a quick pveperf so you see that the server shouldn't be the root-cause
Code:
pveperf
CPU BOGOMIPS:      45598.56
REGEX/SECOND:      1076158
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    559.72 MB/sec
AVERAGE SEEK TIME: 3.99 ms
FSYNCS/SECOND:     4390.28
DNS EXT:           50.07 ms
DNS INT:           3.16 ms

if this is wanted by design, is there a manual possibility how i can create qcows the "old" way like in 2.1 with continually growing space?
thanks in advance, keep up the great work
 

rengiared

Active Member
Sep 8, 2010
96
0
26
Austria
yes, as im still in the test phase i did what you ordered ;)

in the case of the "empty" guest, it got better
Code:
INFO: starting new backup job: vzdump 102 --remove 0 --mode snapshot --compress lzo --storage nem --node pvenem01
INFO: Starting Backup of VM 102 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nem/dump/vzdump-qemu-102-2013_02_20-16_51_28.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task 'b16e3f96-7cb2-40ef-a2e3-e679620d211f'
INFO: status: 0% (3671588864/644245094400), sparse 0% (3671588864), duration 3, 1223/0 MB/s
INFO: status: 1% (7356219392/644245094400), sparse 1% (7356219392), duration 6, 1228/0 MB/s
INFO: status: 2% (13544980480/644245094400), sparse 2% (13544980480), duration 11, 1237/0 MB/s
...
INFO: status: 98% (631619846144/644245094400), sparse 98% (631619846144), duration 531, 1159/0 MB/s
INFO: status: 99% (638610178048/644245094400), sparse 99% (638610178048), duration 537, 1165/0 MB/s
INFO: status: 100% (644245094400/644245094400), sparse 100% (644245094400), duration 543, 939/0 MB/s
INFO: transferred 644245 MB in 543 seconds (1186 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 40MB
INFO: Finished Backup of VM 102 (00:09:07)
INFO: Backup job finished successfully
TASK OK

in the case of the light filled guest it gets worse
Code:
INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --storage nem --node pvenem01
INFO: Starting Backup of VM 101 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nem/dump/vzdump-qemu-101-2013_02_20-17_04_49.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task 'e7c7ede0-f719-4c1e-b6a9-27bbc3a23d40'
INFO: status: 0% (452329472/644245094400), sparse 0% (6426624), duration 3, 150/148 MB/s
INFO: status: 1% (6526992384/644245094400), sparse 0% (56958976), duration 84, 74/74 MB/s
INFO: status: 2% (13123977216/644245094400), sparse 0% (4262871040), duration 122, 173/62 MB/s
...
INFO: status: 98% (631471341568/644245094400), sparse 95% (612637216768), duration 1437, 510/0 MB/s
INFO: status: 99% (638118330368/644245094400), sparse 96% (619284205568), duration 1450, 511/0 MB/s
INFO: status: 100% (644245094400/644245094400), sparse 97% (625410965504), duration 1462, 510/0 MB/s
INFO: transferred 644245 MB in 1462 seconds (440 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 8.50GB
INFO: Finished Backup of VM 101 (00:24:26)
INFO: Backup job finished successfully
TASK OK

and one thing i noticed was the following:
when i started the backup both machines were stopped, and with 2.3 the machine got started but stayd black in the console
i'm going to do tomorrow more tests in this behaviour
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,522
908
163
..and one thing i noticed was the following:
when i started the backup both machines were stopped, and with 2.3 the machine got started but stayd black in the console

as the new backup code is in KVM, even stopped VM´s must be started to run a backup (but they do not boot).
 

rengiared

Active Member
Sep 8, 2010
96
0
26
Austria
ah, thx, good to know
i think i have to try a backup-szenario with a full filled guest, cause the rising time from 5 minutes to 25 is heavy (at the test with 30gb in the guest)
is there a chance that this will speed up a little bit in final? (i'm thinking of my other PVE Hosts where i backup every night about compressed 170gb in about ~2,25hours, and i would get a problem if the backup would take 3 or 4x of the time)

and do you have an approximate date for the release? maybe in the next 2 weeks?
thanks
 

rengiared

Active Member
Sep 8, 2010
96
0
26
Austria
in the meantime i did a test with a better filled guest and i can't complain
Code:
INFO: transferred 644245 MB in 7119 seconds (90 MB/s)
INFO: archive file size: 231.79GB
INFO: Finished Backup of VM 101 (01:58:43)

but with pvetest2.3 a new unknown pci device appeared in the device manager
PCI Slot 3 (PCI bus 0, device 3, function 0)
PCI\VEN_1AF4&DEV_1002&SUBSYS_00051AF4&REV_00
do you have any clue what this could be (2k8r2 as guest-os), has this really to do with memory balooning even if i haven't activated it?
Code:
boot: cd
bootdisk: sata0
cores: 4
ide2: none,media=cdrom
memory: 24576
name: Salzburg01
net0: e1000=36:08:86:66:65:B5,bridge=vmbr0
ostype: w2k8
sata0: local:101/vm-101-disk-1.qcow2,size=400G
sata1: local:101/vm-101-disk-2.qcow2,size=200G
sockets: 2
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!