Question about backing up KVM's with iSCSI storage?

[tom]

I assume most of the difficulty here has been language-related.

no, I clearly understand your words.

I do appreciate the proxmox people trying to help. The scenario I used is from here:

http://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Network_Backing

getting better, now you link to the right section, last time you just linked to the very generic page.

I did exactly as it says. Create iscsi target on the NAS. Create the iscsi target using that in the Storage menu in proxmox. Add the LVM group in the storage menu in proxmox, using the iscsi target just created. We now have some storage. Create a disk in a KVM, and for the backing storage, specify the LVM we just created. It will only allow a rawfile (for reasons I now understand.) Even if the newly created disk is completely pristine, backing up the KVM will use a huge amount of storage for the lvm/iscsi disk, due to the underlying physical disk having random data in it. Does this make more sense?

You did it right and it works. As posted above I already build up a Openfiler test setup and I just can say that it works here, the backup file is just as big as the data on the block device, as expected.

So one step back: is there any open question? can you reproduce the issue on your side? if yes, how can WE reproduce it?

 

[dswartz]

Okay, this is why I am confused. I just reproduced this yesterday again. I plugged a 160GB USB HD into my NAS, made it an iscsi target, etc... Problem happens again. You said (if I recall) that proxmox has to scan the hard drive to see what is real data (I assume it skips zero blocks? e.g. sparse?) In this case, if the physical drive was used for something else, it will have all kinds of random non-zero data on it, so the 'scan for non-zero blocks' will return results indicating lots to copy? I admit I am puzzled as to why it did not happen for you?

 

[tom]

To make sure that I understand what you are doing: you take a used hard disk with for example 100 GB data on it. You attach this to your SAN (openfiler) and create a iSCSI target for the use on Proxmox VE. After added this storage for the use with LVM (virtual disks) you add a 32 GB virtual disk from this pool for your KVM guest. Inside the guest, you format the disk and save, lets say 4 GB of data on it. So we have a 32 GB virtual disk with just 4 GB data on it.

Therefore the backup with vzdump should give you a 4 GB backup tar file (and of course bigger if you include other disks).

 

[dswartz]

Yes, that the precise scenario, but it doesn't work that way Here is why (I think). Yes, I did format that disk with a filesystem, but no 'mkfs' command writes to every block on the disk in question, just scatters inodes, bitmaps and other structures various places. The problem is: the vzdump program (or whatever runs underneath it?) has no idea of filesystems, as far as I can tell - it just is looking at the raw disk, so it sees many GB of non-zero data and so it copies a lot of data to the backup image. I thought I remembered you saying that you have to check for what needs to be copied, but didn't say exactly how that happens. For a sparse raw disk, or a qcow2 disk, it is easy to see the 'real data', but not if the underlying storage is a physical (or network-backed) disk.

 

[dietmar]

The problem is: the vzdump program (or whatever runs underneath it?) has no idea of filesystems, as far as I can tell - it just is looking at the raw disk, so it sees many GB of non-zero data and so it copies a lot of data to the backup image.

Yes, that is the way it works.

 

[dswartz]

So then maybe the wiki should be updated to indicate that the lvm on iscsi storage is problematic for making a vzdump? Am I able to do that?

 

[dietmar]

So then maybe the wiki should be updated to indicate that the lvm on iscsi storage is problematic for making a vzdump?

You will have the same problem with all other formats (just a little bit later). I would not call it a problem - it is the way block level backups work.

 

[dswartz]

Sorry, I was unclear. The typical choices for local disk are qcow2 and raw, neither of which should have this issue - the network backed storage uses a block level backup, so if some user goes from local to network storage, he will see this. Maybe problem is not the right word, but it is not intuitively obvious if one doesn't know what is going on, so maybe a caveat would be in order? Up to you, of course

 

[dietmar]

Sorry, I was unclear. The typical choices for local disk are qcow2 and raw, neither of which should have this issue.

Both format have the same issue! (unless you use a filesystem which write zeros to freed space inside the VM - not sure if such filsystem exists)

 

[dswartz]

I don't believe that is the case though. By definition, qcow2 and raw are sparse file formats. When you format a filesystem on one of these, the mkfs code seeks to various block positions to write inodes and bitmaps and such. Any intervening blocks are not written, and therefore for a sparse format, reading the file will give you intervening blocks of zeroes. Or am I misunderstanding something?

 

[dietmar]

Or am I misunderstanding something?

You miss the fact that the guest will write data. Although the file is sparse when created, it is likely to get written with data over time (resulting in a non-sparse file).

 

[dswartz]

No, I understand that. I thought you were meaning that even a sparse file (qcow2 or raw) would have the same exact problem from the very beginning. What confused me was creating a brand-new disk with only a trivial amount of data, but seeing the backup image huge. I think this all makes sense - thanks for your patience...

 

[e100]

To make your backups smaller zero the unused space in your virtual disk and use vzdump with compression.

If your VM is windows use "sdelete -c" to write zeros to all of the disk free space.
sdelete is available from Microsoft:
http://technet.microsoft.com/en-us/sysinternals/bb897443

In Linux use a command similar to this:
dd if=/dev/zero of=/path/to/mount/point/zero.tmp; rm -f /path/to/mount/point/zero.tmp

I typically perform the above actions right after creating a new virtual machine, especially if it was created on a volume that already had data.