Directory vs LVM Storage for VMs - Pros and Cons?

[Whitterquick]

Hey hows it going yall,

I have read up on the differences between the two and see the the differences between Block level and File level storage. I also see that Snapshots are possible with Directory storage when using the Qcow2 format. When creating VMs and virtual disks on Directory storage I notice it doesn't preallocate the size of the disk; is there any way to enable this or is this considered a good feature (as it can be used for over-provisioning etc)? In general are there any advantages LVM would have over Directory? I see LVM vs LVM-Thin being compared everywhere but not much discussion around the default Directory storage.

Interested to hear some comments on this

 

[Dunuin]

In general thin-provisioning is a good thing. You loose a tiny bit of performance (because blocks need to be allocated dynamically on the fly) but you waste way less space. Lets say I got 20 VMs where each VM needs 50GB for the OS but I also want 50GB free as a spare so the VMs won't run out of space. If i store these 20 VMs on LVM they would need 20x 100GB = 2TB.
Same VMs on LVM-thin just need 1TB and if I would got a 2TB HDD I could easily put 30 of such VMs on that with still 500GB free that these 30 VMs could use if they need more space. And if you later see that these VMs even need more space you can easily add another SSD as a second VM storage and migrate some of them over to the new VM storage. You just need to make sure to monitor your storages so they never get full. But for that there are mlnitoring tools like zabbix, checkmk and so on.
And in case of VMs thin-provisioned storages are way more convinient. I dont know if you ever needed to extend the disk of one of your virtual disks, but it is very annoying. You first need to extend the virtual disk, then rebuild the partition table inside the guest, then extend the VGs, then estend the LVs, then extend the filesystem.
With thin-provisioning I dont need to be thrifty. If I think my VM will need 50GB I can give it a 200 GB disk so it got plenty of space to grow and I possibly dont need to extend it later because of all the spare space. Without thin-provisioning I couldn't do that because I would waste too much space. There I would give the VM 75 or 100GB and it is way more likely that this VM runs out of space so that I would be forced to manually extend the disks. Its way easier to extend your VM storage once every several years instead of extending dozens of virtual disks.

And for the directory vs LVM:
If you use LVM instead of a directory you can save a additional layer in between that adds overhead. With directory you get a additional filesystem layer and filesystems can add alot of overhead. Lets say for example I write 1TB of data as 4k random sync writes. For every 1 TB of data this will add 4TB of metadata so 5TB is written. That basically means your SSD will die 5 times faster and the performance will drop by 80%. A block storage doesn't add this overhead so a virtual disk on a block storage like LVM should be faster than a directory storage.

 

[Whitterquick]

And for the directory vs LVM:
If you use LVM instead of a directory you can save a additional layer in between that adds overhead. With directory you get a additional filesystem layer and filesystems can add alot of overhead. Lets say for example I write 1TB of data as 4k random sync writes. For every 1 TB of data this will add 4TB of metadata so 5TB is written. That basically means your SSD will die 5 times faster and the performance will drop by 80%. A block storage doesn't add this overhead so a virtual disk on a block storage like LVM should be faster than a directory storage.

Oh wow, no wonder my first SSD died with unexplainably little use!

Are there use cases where Directory may be better than LVM? Did you not say LVM-Thin is a little more transparent in terms of security? If I already have a VM setup on Directory how easy is it to back it up, erase the Directory storage, and then re-set it up as LVM/Thin and restore the VM onto it? Are here any issues if the VM was previously on a different type of storage?

Thanks for the explanation, no one seems to talk about Directory much…

 

[Dunuin]

Oh wow, no wonder my first SSD died with unexplainably little use!

Overhead shouldn't be that bad in real life, because async and bigger writes cause way less overhead. But write amplification multiplies with each layer so you get an exponential growth and even small overhead can add up. So its always a good idea to remove a unnessesary layer if possible.

Are there use cases where Directory may be better than LVM? Did you not say LVM-Thin is a little more transparent in terms of security?

Yes, thin-provisioning is always not that secure on an encrypted disk because you can see whats empty and what encrypted. But all your data is still encrypted, so thats secure enough for me.

If I already have a VM setup on Directory how easy is it to back it up, erase the Directory storage, and then re-set it up as LVM/Thin and restore the VM onto it? Are here any issues if the VM was previously on a different type of storage?

VMs don't care about the storage as a backup restore will create a new VM from scratch based on your backupped data. So backup and restore to a new storage should be fine. But you should consider first if you want to loose the qcow2 features. Because if I remember right qcow2 can't be used with LVM so you are loosing the snapshot functionality.

 

[Whitterquick]

@Dunuin when backing up LVM-thin does it take up the full quota or just used space?

 

[Dunuin]

Backups should always only use the size of the actual data. No matter if you use LVM-Thin or LVM because the compression will compress the empty areas.

 

[Whitterquick]

If I already have existing encrypted LVM setup on a disk and I want to use the 'LVM' storage type in Proxmox GUI, is that possible? When I try to setup anything other than 'Directory' the disk is already showing as full, I guess because the underlying LVM is taking up the full disk...

 

[Whitterquick]

Scratch that last question, it doesn’t even make sense