Backup/Restore issue

[michu]

Hi,

I have issue with Backup/Restore beetween Proxmox 1.x and 2.x
Restore image created by Proxmox 2.x is terrible slow when comparing to image created by Proxmox 1.x

Here is example:

1) Proxmox 2.1 restoring machine 115 which backup was created by Proxmox 1.9

extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/1-3/vzdump-qemu-115-2012_09_18-18_02_02.tgz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/115/vm-115-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:115/vm-115-disk-1.raw'
restore data to '/var/lib/vz/images/115/vm-115-disk-1.raw' (12884901888 bytes)
12884901888 bytes copied, 111 s, 110.70 MiB/s

The machine has one disk of size 12G, which is filled in 10% (about 1.2G).
As you can see it's pretty fast (enough for me).

2) Proxmox 2.1 restoring machine 181 which backup was created by Promox 1.9

extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/4-6/vzdump-qemu-181-2012_09_13-17_38_01.tgz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/181/vm-181-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:181/vm-181-disk-1.raw'
restore data to '/var/lib/vz/images/181/vm-181-disk-1.raw' (25769803776 bytes)
25769803776 bytes copied, 460 s, 53.43 MiB/s
extracting 'vm-disk-virtio1.raw' from archive
Formatting '/var/lib/vz/images/181/vm-181-disk-2.raw', fmt=raw size=32768
new volume ID is 'local:181/vm-181-disk-2.raw'
restore data to '/var/lib/vz/images/181/vm-181-disk-2.raw' (26843545600 bytes)
tar: write error
26843545600 bytes copied, 161 s, 159.01 MiB/s

Once again restore is pretty fast, this machine had 1x24G and 1x25G disk.

2) Proxmox 2.1 restoring machine 126 which is backup created by the same Proxmox 2.1

extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/1-3/dump/vzdump-qemu-126-2012_09_18-18_54_24.tar.gz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/126/vm-126-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:126/vm-126-disk-1.raw'
restore data to '/var/lib/vz/images/126/vm-126-disk-1.raw' (34359738368 bytes)
34359738368 bytes copied, 1361 s, 24.08 MiB/s

This machine has 1x32G disk, restore was very slow (1361s) :/

----------------

In all situations backups are restored from SSH share (fuse.sshfs) to local share (ext3).

Here is my pveversion:
pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-74
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-14-pve: 2.6.32-74
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-49
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-31
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-8
ksm-control-daemon: 1.1-1

And here is my pveperf:
CPU BOGOMIPS: 92581.60
REGEX/SECOND: 1431166
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 567.44 MB/sec
AVERAGE SEEK TIME: 6.04 ms
FSYNCS/SECOND: 5977.17
DNS EXT: 83.11 ms
DNS INT: 0.39 ms

Do you have any ideas ? We're just moving from 1.x to 2.x with new server and new clean installation. For now our biggest problem is restore time - it's unacceptable :/


Regards,
michu

 

[dietmar]

Is that reproducible, or only happened one time?

 

[michu]

Is that reproducible, or only happened one time?

Hi Dietmar,

It seems it's reproductible. I backed up machine 116 on Proxmox 2.1 and then restored.
Here is task output:

extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/4-6/dump/vzdump-qemu-116-2012_09_18-21_00_31.tar.gz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/116/vm-116-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:116/vm-116-disk-1.raw'
restore data to '/var/lib/vz/images/116/vm-116-disk-1.raw' (12884901888 bytes)
12884901888 bytes copied, 498 s, 24.67 MiB/s
TASK OK

Few days ago I restored this machine from Proxmox 1.x
Here is task output:

extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/4-6/vzdump-qemu-116-2012_09_13-15_56_02.tgz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/116/vm-116-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:116/vm-116-disk-1.raw'
restore data to '/var/lib/vz/images/116/vm-116-disk-1.raw' (12884901888 bytes)
12884901888 bytes copied, 34 s, 361.41 MiB/s
TASK OK

The machine has not changed inside. It's DNS server so it don't write much to HDD (logs only in /var/log).

IMHO something changed in your backup method/algorithm or maybe tar/gzip. I found that Proxmox 2.x backups are larger 10%-30% than backups from Proxmox 1.x.
In both situations I used GZIP as compression method.


Regards,
Michu

 

[michu]

I have one suggestion:

1) I see that Proxmox 1.x created files *.tgz, so I assume that it used tar with option --gzip, additionally I assume that you used --sparse flag to handle sparse files (raw format).
2) I see that Proxmox 2.x created files *.tar.gz, so I assume that you first create tar archive (with --sparse flag) and then you gzip, maybe there is problem with handling this sparse files... So in result files created by Proxmox 2.x doesn't handle sparse region of files and decompress/restore is much slower (system has to restore whole file instead of part of this file).

Regards,
Michu

 

[tom]

use lzo, much faster.

 

[michu]

use lzo, much faster.

Tom, this is not an option for us - lzo takes more space, we have many machines which we backup few times in week. In fact we need much space for it. Our space for backup is filled in 70%-80%.
It looks like bug in Proxmox 2.x. Explain me why backups created with Proxmox 1.x compressed with gzip are smaller and restore multiple times faster (34s against 498s - thats 15x faster !!!) than backups created in Proxmox 2.x ? I use same Proxmox 2.x to restore backups (same host, version, utilities, etc) and simply 1.x backups restore multiple times faster. I suggested where the problem can exists...

IMHO that issue should be investigated and fixed if possible, it's reproductible so at least it's worth to check why this is happening.


Regards,
Michu

 

[michu]

Tom,

I tried lzop as you suggested. Using Proxmox 2.x I created backup of machine with 2 HDDs (24g and 25g), backup process was very fast, lzo did good job.
But... still comparing restore time to gzip backup created with Proxmox 1.x is worse....
I restored both backups on same Proxmox 2.x.

GZIP with Proxmox 1.x:
extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/4-6/vzdump-qemu-181-2012_09_13-17_38_01.tgz'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/181/vm-181-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:181/vm-181-disk-1.raw'
restore data to '/var/lib/vz/images/181/vm-181-disk-1.raw' (25769803776 bytes)
25769803776 bytes copied, 460 s, 53.43 MiB/s
extracting 'vm-disk-virtio1.raw' from archive
Formatting '/var/lib/vz/images/181/vm-181-disk-2.raw', fmt=raw size=32768
new volume ID is 'local:181/vm-181-disk-2.raw'
restore data to '/var/lib/vz/images/181/vm-181-disk-2.raw' (26843545600 bytes)
tar: write error
26843545600 bytes copied, 161 s, 159.01 MiB/s
TASK OK

LZOP with Proxmox 2.x:
extracting archive '/var/lib/vz/pvesshfs/pve1x02/daily/4-6/dump/vzdump-qemu-181-2012_09_19-14_22_51.tar.lzo'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio1.raw' from archive
Formatting '/var/lib/vz/images/186/vm-186-disk-1.raw', fmt=raw size=32768
new volume ID is 'local:186/vm-186-disk-1.raw'
restore data to '/var/lib/vz/images/186/vm-186-disk-1.raw' (26843545600 bytes)
26843545600 bytes copied, 255 s, 100.39 MiB/s
extracting 'vm-disk-virtio0.raw' from archive
Formatting '/var/lib/vz/images/186/vm-186-disk-2.raw', fmt=raw size=32768
new volume ID is 'local:186/vm-186-disk-2.raw'
restore data to '/var/lib/vz/images/186/vm-186-disk-2.raw' (25769803776 bytes)
tar: write error
25769803776 bytes copied, 569 s, 43.19 MiB/s
TASK OK

So it's 621s (proxmox 1.x, gzip) against 824s (proxmox 2.x, lzop).
Additionally I found next issue, my disks was replaced disk-2.raw become virtio0, and disk-1.raw become virtio1. I'm just perfectionist so badly ordered things looks terrible for me
Most interesting thing is that backup compressed with proxmox 2.x lzop is smaller 10.98GB vs 11.42 GB in proxmox 2.x gzip.

Regards,
Michu

 

[tom]

your examples are confusing for me, pls define a simple and easy reproducible case and and I try to reproduce it here.

 

[michu]

Tom,

OK, here is path to test:

1) Create VM in Proxmox 1.9 and install for example Ubuntu 10.04 (my machine has this system). Use virtio with raw format file. Size 32G for example.
2) In Proxmox 1.9 backup this machine on shared disk, in my situation it's fuse.sshfs share.
3) In Proxmox 2.1 restore this VM from shared disk, note restore time, transfer speed etc.
4) In Proxmox 2.1 backup this machine again on the same shared disk.
5) In Proxmox 2.1 restore this VM from shared disk, again note restore time, transfer speed etc. You will see that it's multiple times slower than restore machine backed up via Proxmox 1.9.


Regards,
Michu

 

[tom]

you test with LVM snapshot backup, or just offline.?

 

[michu]

Tom,
In this case it was stop machine backup.
Is there any difference beetween stop and snapshot backup image ???

Regards,
Michu

 

[dietmar]

To avoid further confusion, we optimized vzdump in 2.X for backup speed. So we do not scan for sparse files when compression is used. I guess that explains the behavior.

 

[dietmar]

5) In Proxmox 2.1 restore this VM from shared disk, again note restore time, transfer speed etc. You will see that it's multiple times slower than restore machine backed up via Proxmox 1.9.

1.) This is only true if you use comoression - without compression you get exactly the same speed?

2.) This depends if the image is sparse. Backup and restore with lzo is much faster now when the image does not contain many holes.

 

[michu]

Dietmar,

1) I haven't tried without compression. Since I update on network storage avoid compression will be huge loose on network bandwidth (it's rather slower than hdd),
2) This night I created backups using lzo, the image sizes are nearly same as gzip (about 1-3% difference in sizes, strange ?), backup with lzo took 25 minutes vs 1h with gzip so it's good, but restore is still slower than restoring sparse files...

IMHO scanning for sparse files should be configurable via backup job definition as additional option for raw files. In my situation restoration time is always more important than backup time. I have two servers, but without shared storage so if one of servers die, I move machines to another server. In this situation my users wait for it, so it's better to have faster restore than backup. The good information is that servers don't die so often


Regards,
Michu

 

[dietmar]

IMHO scanning for sparse files should be configurable via backup job definition as additional option for raw files.

The plan is to replace 'tar' format with a better format which can handle sparse files effectively.