[SOLVED] Large backup file-size for small VMs - why & workarounds?

[Razva]

Hey,
I have a few KVM VMs to which I've allocated 50gb of storage. From those 50gb only 5gb are currently used, so the "real" disk usage is 5gb. Still, generated backups reach the ~50gb mark, not the ~5gb mark. I guess that Proxmox makes an image of the...unused space? If yes, is there any way to workaround this?
Thank you

 

[wolfgang]

Hi,

do you use compression?

 

[Razva]

do you use compression?

Yup, GZIP. The partition on which the VMs are stored is ext4 @ LVM-Thin.

 

[zima]

use fstrim in vm and use virtio iscsi with discard

 

[LnxBil]

use fstrim in vm and use virtio iscsi with discard

If that does not work, just overwrite the free space of your disk with zeros. On linux, do it with dd, on Windows you can use ccleaner.

 

[spirit]

From those 50gb only 5gb are currently used, so the "real" disk usage is 5gb.

Are you sure that only 5gb of BLOCK are used ?
If you write a big file of 50gb, then you delete it, they are still 50gb block allocated.

you need to use fstrim/discard or write zeros to really free space.

(Proxmox backups are at block level)

 

[Razva]

use fstrim in vm and use virtio iscsi with discard

Mmmmm...VirtIO or SCSI? Please check this out:

Are you sure that only 5gb of BLOCK are used ? If you write a big file of 50gb, then you delete it, they are still 50gb block allocated. you need to use fstrim/discard or write zeros to really free space.

I write 50gb for a couple of times a day and, at the end of the day, everything gets removed. So I suppose that I'm still having 50gb block allocated when the backup occurs?

Sorry for my lack of knowledge, but isn't fstrim for SSDs? Or am I misplacing trim with fstrim? Do you have any tutorial/vid about setting up fstrim? Note: I'm using LVM.

 

[LnxBil]

Sorry for my lack of knowledge, but isn't fstrim for SSDs? Or am I misplacing trim with fstrim? Do you have any tutorial/vid about setting up fstrim? Note: I'm using LVM.

It only works if you have an SSD below the LVM or you use thin-provisioning. Best option is to use SCSI as Bus/Device and VirtIO SCSI as controller type and enable discard. If you backend storage supports discard, it'll success, otherwise it'll fail and you end up with zeroing your free space, which always works, but needs more time and increases the load on you storage backend dramatically with respect to just issue some TRIM commands to the controller.

 

[spirit]

It only works if you have an SSD below the LVM or you use thin-provisioning.

No, it's works with hdd too. But it's not work with nfs for example. (trim command need to go to the storage, and nfs don't support it for example)

 

[LnxBil]

No, it's works with hdd too. But it's not work with nfs for example. (trim command need to go to the storage, and nfs don't support it for example)

Any case with LVM not involving thin provisioning (as I already stated), please tell me.

 

[spirit]

Any case with LVM not involving thin provisioning (as I already stated), please tell me.

I think it should work (it should remove/trim block, but don't shrink lvm volume as it's not thin provisionned), but you need to enable discard support.

cat /etc/lvm/lvm.conf


# [...]
devices {
# [...]
issue_discards = 1
# [...]
}
# [...]

 

[Razva]

I think it should work (it should remove/trim block, but don't shrink lvm volume as it's not thin provisionned), but you need to enable discard support./QUOTE]
For the sake of not destroying our data, can you please let us know what commands/information should we post in order offer a full overview of our servers?

 

[Razva]

You can find the solution here.

Long story short:

• use ZFS as your storage
• use VirtSCSI as "Bus/Device" when creating your VM
• setup TRIM in the OS (it's not installed by default on CentOS / Debian / Ubuntu) or DISCARD. Note that "fstrim" needs to be manually executed so you should set a daily cron task that runs before doing the backups

I hope this helps.

 

[aderumier]

• Note that "fstrim" needs to be manually executed so you should set a daily cron task that runs before doing the backups

.

Note that in recent kernel (>= 4.7), "discard" mount option is now async, so you can enable it without write penality. (and no more fstrim cron is needed)

 

[Razva]

Note that in recent kernel (>= 4.7), "discard" mount option is now async, so you can enable it without write penality. (and no more fstrim cron is needed)

In this /etc/fstab file, which is similar on most CentOS / Debian / Ubuntu VMs with ext4, where should the "discard" option be placed?

/dev/mapper/mx--vg-root /               ext4    errors=remount-ro 0       1
UUID=431c7593-3b37-4bde-9c22-e97e27bc49e4 /boot           ext2    defaults        0       2
/dev/mapper/mx--vg-swap_1 none            swap    sw              0       0

What about these xfs volumes?

/dev/mapper/centos_web1-root /                       xfs     defaults        0 0
UUID=538887fb-a252-4a2e-8f9e-5c95cdc24098 /boot                   xfs     defaults        0 0
/dev/mapper/centos_web1-swap swap                    swap    defaults        0 0

 

[elamigosam]

I know this is an old thread, but I am having the same issue big backups.
Do we still need to do manual fstrim?

I was running it on my ubuntu servers 18.04 and i do get an answer back

/: 274.3MiB (287625216 bytes) trimmed

 

[Razva]

I know this is an old thread, but I am having the same issue big backups.
Do we still need to do manual fstrim?

I was running it on my ubuntu servers 18.04 and i do get an answer back

/: 274.3MiB (287625216 bytes) trimmed

The most simple solution is to setup a cron on the guest VM, something like

*/30 * * * * fstrim --all >/dev/null 2>&1

 

[elamigosam]

The most simple solution is to setup a cron on the guest VM, something like

*/30 * * * * fstrim --all >/dev/null 2>&1

Thank you.

Do we still need to enable Discard check box on the Hard Disk options in proxmox?