Thick Provisioning to Thin Provisioning
- Thread starter repa
- Start date
if your storage support discard, you could enable discard on the your vm disk and do an fstrim.
or you can backup/restore
(Note that raw format is thin provisionned too by default, just do a "du -sh myfile.raw" to see the real space usage)
or you can backup/restore
(Note that raw format is thin provisionned too by default, just do a "du -sh myfile.raw" to see the real space usage)
We're migrating from local storage to CEPH.
rbd du -p cephpool1
NAME PROVISIONED USED
vm-100-disk-0 32 GiB 32 GiB
vm-102-disk-0 40 GiB 40 GiB
vm-104-disk-0 41 GiB 41 GiB
vm-107-disk-0 72 GiB 72 GiB
vm-108-disk-0 41 GiB 41 GiB
Backup / Restore is not an option, as we're talking about 300 vm's.
rbd du -p cephpool1
NAME PROVISIONED USED
vm-100-disk-0 32 GiB 32 GiB
vm-102-disk-0 40 GiB 40 GiB
vm-104-disk-0 41 GiB 41 GiB
vm-107-disk-0 72 GiB 72 GiB
vm-108-disk-0 41 GiB 41 GiB
Backup / Restore is not an option, as we're talking about 300 vm's.
enable discard on the vm disk, then use "fstrim -a" in your linux vms. (for windows they are a schedule task in "optimize disk" tool)
(before the migration, it was possible to enable agent option: "fstrim", to do it for you after disk migration)
(before the migration, it was possible to enable agent option: "fstrim", to do it for you after disk migration)
That depends on your target storage. If you e.g. migrate from RAW file to QCOW2 file, it is possible and all the fstrim magic @spirit wrote about can be used to free up used blocks that are actually free. It also works if you migrate to ZFS, but as you mentioned CEPH (as I read now), the rados device you will use internally, is also thin provisioned, so that should work, too.
IIRC, you can do each storage migration from the cli, so that you only need to loop over your VMs.
But #3 spirit said: "Note that raw format is thin provisionned too by default", I need disable thin provision
Recreate the RAW file as thick provisioned with dd after creating it or convert it via qemu-img manually (google is your friend). There is no way to do this in PVE itself except with thick LVM and ZFS without the 'sparse' check mark.
What happens if someone imports a thick provisioned vmdk via importer into a thin-provisioned storage (like ZFS or ceph) will the imported disk still be thick or can it become thin somehow? (I will test if no one knows) - havent tried yet.
Last edited:
depends on the storage.
If you import to ZFS for example and the "thin provision" option is enabled, then it will be thin, as in, zeros won't be written*.
On RBD, you would need to do a trim/discard after the import, as, IIRC, on RBD, zeros will be written initially.
Not sure how LVM-thin will behave from memory.
* for this to work, you cannot disable compression, otherwise ZFS will also write all zeros.
If you import to ZFS for example and the "thin provision" option is enabled, then it will be thin, as in, zeros won't be written*.
On RBD, you would need to do a trim/discard after the import, as, IIRC, on RBD, zeros will be written initially.
Not sure how LVM-thin will behave from memory.
* for this to work, you cannot disable compression, otherwise ZFS will also write all zeros.
Note that the ZFS storage
I'm guessing for LVM-Thin it behaves similar to this: https://forum.proxmox.com/threads/v...fter-migrating-it-to-lvm-thin-storage.142070/
Thin provision option affects the refreservation property. Discard/fstrim will still work without it.I'm guessing for LVM-Thin it behaves similar to this: https://forum.proxmox.com/threads/v...fter-migrating-it-to-lvm-thin-storage.142070/
Yea I wondered about that as well, the "Thin Provision" Option in Datacenter -> Storage -> ZFS is kinda confusing, as in both cases (not enabled, enabled) it is thin-provisioned. ZFS Reservation just wont let you create any new disks if reserveration is full. Might be cool, mentioning or even changing the term in the Datacenter -> Storage -> ZFS Definition?
This distinction is irrelevant for PVE or any other virtualization platform that does not use files for its storage, as @aaron already wrote.
I would ALWAYS recommend to run trim/discard inside of the guest to really free up actually free space. This is not so common knowledge as VMware does not really support this in VMDK files and at least at the time of my last test needs offline time to compact the file. Files are so bad for VMs, a lot of performance and space saving potential is just wasted on file-based systems. Even LVM-thin and RBD are not really that good with respect to thin provisioning, as their allocation units are just to darn huge and need further optimization. The allocation units are in the MBs and not 8KB or 16KB depending on your PVE generation in ZFS.
We're just looking at the top level space usage, not the backend with replication / copy / raid /etc. If you have only one bit written (everything else is zero), you have always written one allocation unit, so MBs in LVM-thin, MBs in RBD without compression, maybe less in RBD with compression and in ZFS always the volblocksize (less with compression). This is bad for space savings. Fragmentation of the filesystem also plays into this, so you should always defragment your disk. Having a lot of scattered 4K guest OS filesystem blocks on different huge 2 MB allocation units is a problem and stands in the way of best thin provision experience. Defragmenting is not as easy as it sounds, as most VMs are emulating SSD (optimizing the I/O scheduler to not optimize for disk queue stuff), so that e.g. Windows detects the SSD and then NOT defragment the disk at all. This makes sense for "real" SSDs, but not for highly complex storage systems in virtualization environments with thin provisioning. In of-the-shelf-storage systems, you often have 128K blocks which are better than the MB ones, but still worse than the 8K/16K ones. Just to mention it, bigger allocation units are not always bad, they also have the advantage of much better compression rates, as it's easier to compress larger files than to compress smaller files. This also plays into the backend space usage of storage systems, yet LVM-thin does IMHO not compress stuff per default. RBD does also not compress per default, but can be activated easily. ZFS does compress per default, yet with an ashift value of 12, there is not much compression rates visible. If you're lucks if you're still on 512 native blocksize and ashift 8, you will notice much better compressability.
This topic is a huge rabbit hole. There is one way to optimize for your actual space usage on disk. If you run such optimizations regularly, you will save a lot of space in a thin provisioned environment, yet you will generate much more backend backup storage usage, as a defragmented disk is usually less deduplicable than before as blocks have changed. So this may come back and bite you in the ass.