Making backups small / thin as possible

[locusofself]

Looking for some feedback here, I recently noticed some of my backups were larger than I expected given the amount of actual data residing in the guest VMs. But I realized this is because some of those guest VMs had *previously* stored a lot of data and though I had deleted it it wasn't discarded.


So I believe this is good formula for ensuring backups are "thin" as possible?

1. Make sure to use Virtio SCSI driver (this seems to be default for disk in my 4.4 cluster)
2. Set discard=on in the disk options in proxmox/KVM for the guest
3. Use LVM storage (?) or .qcow2 and not .raw file for disk
4. Run "fstrim -a" or similar within the guest operating system prior to backup running to discard unused blocks from the filesystem. (this can be run through cron or systemd, and is already running in some distros)


Do I have this right? Thanks

 

[locusofself]

Am I correct that this will not work with .raw disks and the backup size will always be almost the same size as the disk file itself even if I fstrim/discard ? I have a bunch of .raw disk

 

[LnxBil]

For raw files, you can use dd to zero out the free space in Linux or sdelete on Windows. This is slow and will write a file with only zeros inside to "level out" every residual data on the disk.

 

[locusofself]

For raw files, you can use dd to zero out the free space in Linux or sdelete on Windows. This is slow and will write a file with only zeros inside to "level out" every residual data on the disk.

I do understand sdelete is used to "secure delete" in windows but that isn't really my objective, my objective is to set up my VMs in proxmox in such a way as to produce a backup that is as small as possible without losing any real data that isnt just garbage from past deleted files or empty blocks.

Looking to see if there are any further clarifications or additions to the process I outlined in the original post

 

[fireon]

Am I correct that this will not work with .raw disks and the backup size will always be almost the same size as the disk file itself even if I fstrim/discard ? I have a bunch of .raw disk

Not with ZFS and LVM-Thin and normal LVM. Maybe this was so in the past. Maybe Raw uses a little pit more...

 

[LnxBil]

I do understand sdelete is used to "secure delete" in windows but that isn't really my objective, my objective is to set up my VMs in proxmox in such a way as to produce a backup that is as small as possible without losing any real data that isnt just garbage from past deleted files or empty blocks.

Looking to see if there are any further clarifications or additions to the process I outlined in the original post

Then look again at the parameter -z which was introduced exactly to wipe free space, hence better compressability of the virtual disk.