backup empty kvm very slow

[emanuelebruno]

Hi all,
why is backing up an empty VM under Proxmox is so slow? Is it possible that no one has ever noticed?

example: whether I create an empty 1 TB qcow file, or I create a 1 TB KVM in a ZFS partition, and install, for example, debian 10 (ssh server only and nothing else), my server with a CPU Intel(R) Xeon(R) CPU E5-2470 take 28 minutes to backup!!

INFO: starting new backup job: vzdump 101 --node s0 --mode stop --storage local2 --remove 0 --compress gzip
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-01-22 09:25:20
INFO: status = stopped
INFO: update VM 101: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: BackupCT
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 1T
INFO: creating archive '/local2/dump/vzdump-qemu-101-2020_01_22-09_25_20.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task '346c984b-d79d-4183-a0fe-f7a6db44bf65'
INFO: status: 0% (593494016/1099511627776), sparse 0% (301903872), duration 3, read/write 197/97 MB/s
INFO: status: 1% (11119230976/1099511627776), sparse 0% (10374496256), duration 22, read/write 553/23 MB/s
INFO: status: 2% (22407675904/1099511627776), sparse 1% (21662392320), duration 39, read/write 664/0 MB/s
INFO: status: 3% (33644544000/1099511627776), sparse 2% (32898711552), duration 56, read/write 660/0 MB/s
INFO: status: 4% (44333989888/1099511627776), sparse 3% (43588120576), duration 72, read/write 668/0 MB/s
INFO: status: 5% (54985621504/1099511627776), sparse 4% (54239203328), duration 88, read/write 665/0 MB/s
[...]
INFO: status: 100% (1099511627776/1099511627776), sparse 99% (1098065768448), duration 1700, read/write 609/2 MB/s
INFO: transferred 1099511 MB in 1700 seconds (646 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 395MB
INFO: Finished Backup of VM 101 (00:28:24)
INFO: Backup finished at 2020-01-22 09:53:44
INFO: Backup job finished successfully
TASK OK

Instead if I unzip the backup it takes less than 30 seconds!

Another example, if I have two ZFS partitions (in two different HDDs), and I want move a near 1 TB empty disk from 1 ZFS partition to another, it take 4 minutes...

Watching with iotop noticed a strange thing:



DISK READ 3,9 G/s , DISK WRITE 3.46 M/s ??? WHY? it's a damn empty file!

 

[Stefan_R]

DISK READ 3,9 G/s , DISK WRITE 3.46 M/s ??? WHY? it's a damn empty file!

Says who? PVE cannot assume that the file is empty, as the VM running can place data at any point within it. So it has to read it all - that's your DISK READ stat. However, since all it's reading is '0's, gzip has a ridiculously high compression efficiency, thus not much needs to be written to disk.

That is the disconnect you're seeing. As to why it takes so long - limiting factors are disk speed (mostly read, as you saw in your case) and CPU efficiency, since the compression algorithm also takes some performance. Try it with lzo compression, or enable pigz for multi-threaded compression (install 'pigz' with apt, then set 'pigz' to 1 in /etc/vzdump.conf).

 

[guletz]

Hi,

As @Stefan_R said, is a problem of read's. Also, it could be in addition some zfs related facts, most probably your metadata could not fit in RAM(so the backup task need to do a lot of seeks on disks for find where are phisicaly data located on disk).

Instead if I unzip the backup it takes less than 30 seconds!

To make a zip archive is one thing, to unzip is a complete new story

WHY? it's a damn empty file!

No. A minimal debian vm is not empty(395 MB archive in your case).

Anyway in case of zfs, it not important how big is vdisk. Important is how many MB are in use.

Good luck / Bafta.

 

[emanuelebruno]

[...]
No. A minimal debian vm is not empty(395 MB archive in your case).
[...]

I understand what you say, but the problem still occurs in other situations. I try to explain.

I know that qcow is a shrinked file (it means that if I create a qcow of 5 TB size for the first time, Proxmox will spend few seconds because it will not allocate 5 TB of empty file of "00" hexadecimal bytes like a raw file, are you agree with me until now? For this reason, you can create whatever you want of empty files even if in reality your proxmox stays in an 120 GB SSD drive...

Ok, now that you have created a 5 TB empty kvm without any kind of operative system, try to do a backup, and tell me how long it will take ...

Although the qcow file is a few KBytes, we can make a bet that the backup will not take a few seconds and this is not normal.

 

[emanuelebruno]

[...]

Thank you for your reply...

please can you read my answer to guletz ?

 

[Stefan_R]

I know that qcow is a shrinked file (it means that if I create a qcow of 5 TB size for the first time, Proxmox will spend few seconds because it will not allocate 5 TB of empty file of "00" hexadecimal bytes like a raw file, are you agree with me until now?

Yes, that's called thin-provisioning. Note that the file still has a size of 5 TB, but only the used space is allocated (i.e. written to disk). This does not mean, that written data is all serialized, that is, data can be written to the end of the file immediately after creation, and not just at the start (like what would happen with an 'expanding' file, instead of a thin-provisioned one).

Ok, now that you have created a 5 TB empty kvm without any kind of operative system, try to do a backup, and tell me how long it will take ...

Probably a while, since PVE has to back up 5 TB of potential data.

As I've already explained, one cannot assume that the file is empty. We have to scan all 5 TB of data space (even if it empty or not even allocated), since a guest OS does not behave like a regular file writer.

What can happen, for example, is that a guest uses, say 100 MB of data at the beginning of the 5 TB drive, but then a user comes along and creates a second partition on the guest's root drive, allocated at exactly 4 TB. They proceed to write 50 MB to that new partition. Now, the qcow2 file will have a size-on-disk of (approximately) 150 MB. However, if we were to only back up the first 150 MB of the file, we would miss the second partition and instead only copy the OS install and 50 MB of blank data.

And yes, simply cp'ing the qcow2 file will be faster, since then the host kernel can fast-path the copying of unallocated blocks, but that breaks down as soon as you want to back up a running VM (as the guest could write to the file during the backup, breaking all sorts of consistency guarantees). So we have to do it this way, and that means that yes, backup speed is partially dependent on the potential file size, not just the actually allocated one.