Centos 7 KVM Backup

[matthew]

I have a number of Centos 7 KVM virtual machines running under Proxmox. They are mixture of XFS and EXT4 file systems. Is there anyway to zero out the unused disk space so the backups of the entire KVM done with Proxmox are not so large?

 

[LnxBil]

Depends on the storage if you can use fstrim, but dd'ing does always work, e.g. running this inside of each filesystem-based mountpoint:

dd if=/dev/zero of=zero bs=64k; sync; sync; sync; rm -f zero

 

[matthew]

How do I make fstrim work?

This is a proxmox install with 2 sata drives in zfs RAID1 like array. Thin Provision is enabled on the Image storage drives. I used VirtIO SCSI when creating the VMs. I have one Centos7 install that is XFS filesystem and another that is EXT4. The compressed backups on both are over twice there actual disk usage.

 

[LnxBil]

How do I make fstrim work?

https://pve.proxmox.com/wiki/Qemu_trim/discard_and_virtio_scsi

 

[matthew]

https://pve.proxmox.com/wiki/Qemu_trim/discard_and_virtio_scsi

I managed to make fstrim work by changing:

This line:
virtio0: local-zfs:vm-101-disk-1,size=1000G
To:
scsi0: local-zfs:vm-101-disk-1,size=1000G
In:
/etc/pve/nodes/prx10/qemu-server/101.conf

Now it wont boot unless I select the boot device on the console at start up. It then works fine. This is Centos 7 with ext4.

Any ideas?

Thanks.

 

[matthew]

I resolved the boot issue. I missed bootdisk: in the conf file. Also, had to add discard=on too scsi0 line.

I have a guest centos 7 KVM using LVM. How do I get TRIM to work with LVM?

 

[LnxBil]

I resolved the boot issue. I missed bootdisk: in the conf file. Also, had to add discard=on too scsi0 line.

Yes, you can also change all the things in the GUI, just remove the disk (it is still there), add it again and change the bootdisk-Option. This is red if you do not have e.g. virtio0, so this is a bit more "error showing". But CLI also works fine.

I have a guest centos 7 KVM using LVM. How do I get TRIM to work with LVM?

It only works if you used LVM thin provisioning.

 

[matthew]

Yes, you can also change all the things in the GUI, just remove the disk (it is still there), add it again and change the bootdisk-Option. This is red if you do not have e.g. virtio0, so this is a bit more "error showing". But CLI also works fine.



It only works if you used LVM thin provisioning.

If its a regular LVM and not thin is there anyway to make fstrim work to reduce physical disk usage and backup size?

 

[dietmar]

If its a regular LVM and not thin is there anyway to make fstrim work to reduce physical disk usage and backup size?

No, plain LVM is pre-allocated.

 

[matthew]

This is a Centos 7 install with EXT4 on top of LVM. Is there any easy way to convert it from LVM to thin-LVM or no LVM?

 

[LnxBil]

Is there any easy way to convert it from LVM to thin-LVM or no LVM?

Sure, use the "move disk" command in PVE if you have thin-LVM (which is not so thin as you might think).

reduce physical disk usage and backup size?

Sure, use something like this for each "real" filesystem:

dd if=/dev/zero of=/zero bs=64k; sync; sync; sync; rm -f /zero

 

[matthew]

Sure, use the "move disk" command in PVE if you have thin-LVM (which is not so thin as you might think).

This is not thin-LVM just LVM.

 

[matthew]

dd if=/dev/zero of=/zero bs=64k; sync; sync; sync; rm -f /zero

[/QUOTE]

Anyway to do this but have it stop when the filesystem is 95 percent full?

 

[LnxBil]

This is not thin-LVM just LVM.

It does not matter what you currently have. To change it, you have to use the "move disk" to a storage that is right for you. So just create a thin pool and move the stuff around.

Anyway to do this but have it stop when the filesystem is 95 percent full?

It normally fills very quickly and should return in a few seconds. I you have multiple TBs of free space, you should have probably started with less.

 

[matthew]

So when I setup my Centos 7 VM under proxmox. What is the difference between if I used thin-KVM vs regular-KVM on the Centos 7 boxes root and largest partition? This regular-KVM does not pass through the fstrim discards from the Centos box to the host Proxmox box which in my case uses ZFS?

 

[matthew]

No, plain LVM is pre-allocated.

If I try to use fstrim -av on Centos 7 with its ext4 partition mounted on a standard-LVM running on a proxmox host using ZFS is the worst that will happen is it won't work?

I have tried this on a test VM and it does seem to shrink the backup size running fstrim -a. I downloaded a large ISO with wget on the Centos VM. Deleted it and then did a backup of the KVM in proxmox. I then did fstrim -av and then did the backup again. The backup size decreased by the size of the ISO. This a Centos 7 KVM with its ext4 root partition mounted on a standard-LVM then running inside proxmox using simple ZFS setup.

 

[LnxBil]

ZFS does support TRIM as well as thin-LVM, so this works. You can also run TRIM on other backend storage systems and in the worst case it says that it has freed something, but it actually does not (regular LVM does this).

Your described findings are 100% correct. This is how trim should work with respect to backup. The thin LVM on the other hand is different. You have to consider the minimum block unit that can be freed and in the worst case, this is larger - even much larger - than your block size on your guest file system, so you will most probably not see such a big impact there. Also fragmentation is relevant on storage, but not in the backup process.

Another idea:
If you already have ZFS, why don't you use the qemu-agent to freeze your VM (same as the backup process does), snapshot via ZFS, unfreeze via QEMU and send/receive your differential backup to another ZFS (on another machine) for backup? This yields the same kind of consistency guest storage wise.

 

[matthew]

Thanks for the reply. fstrim seems to create a great deal of system load the first time it is issued after reboot on the KVM client. Would it be better to mount the ext4 or xfs files system in fstab on the centos client with the discard option?

 

[LnxBil]

Would it be better to mount the ext4 or xfs files system in fstab on the centos client with the discard option?

As far as I remember, this is even worse, because each remove I/O takes much longer so that the overall performance is degraded. As always: Try for yourself.

Here is some information about periodic vs. continuous trimming:
https://wiki.archlinux.org/index.php/Solid_State_Drive#TRIM