BTRFS on LUKS on zvol advices

[stckptr]

Hi,
I currently have a ZFS pool of SSDs where I have some VMs on zvols (ext4 fs), and I want to create a VM with an encrypted disk that would be encrypted using LUKS and use BTRFS as a file system. I have read that using a COW filesystem (BTRFS) on a COW filesystem (ZFS) is not a good idea performance wise, and I'm wondering why ?
From my understanding, if I want to modify a file in BTRFS, it will allocate a new space to write the data (due to the COW natures of BTRFS) which to the underlying ZFS will look like a new file. Then BTRFS will discard the old location, which will be freed on the zvol. To me, it doesn't seem to be more total work (in terms of read/writes) apart from the cost of the two filesystems (and luks encryption) code to execute? Am I understanding it wrong ?

 

[Fantu]

Cow filesystem (BTRFS and ZFS) basically have a write amplification itself, with cow on cow even worse because you add it.
But the problem is not only that, there is also a greater fragmentation, on hdd it is very relevant but also on ssd it is not to be underestimated because on large files (like vm disks) with many writings (even if small) you would have a medium-long term big fragmentation for which even apparently small and fast operations could end up having a significant weight even on ssd, and to solve it you should resort to defragmentation or complete rewriting of the file.
Regarding the increased space consumption it can depend on different factors.
For example one is related to snapshots, without snapshots I suppose it is the best case where you have less and you can free up more space when delete something, but if you start having snapshots on one or both sides you can have a bigger impact on the space too.

It is advisable to avoid such cases and if you want to use fs cow (like zfs or btrfs) on vm disks on the host it is better to have without fs (for example on lvm) or on non cow fs, it is also important to note the file format used for vm disks, if you use qcow2 it is also an additional cow layer.

If you want to use cow on cow anyway then you will have to pay more attention to disk wear, to the space occupied and the performance.

I don't use vm disks on fs cow after I saw during tests about ten years ago on btrfs that it is not convenient for me in my use cases and with the hw available, but I could notice the impact of cow on cow in other cases. A few years later I also tried anyway on zfs but discarded it there too, as for performance with many optimizations, a lot of ram it was better than btrfs while as for disk wear it seemed even worse than btrfs.

For example recently on a filesystem with btrfs in raid 1 on 2 3TB HDD disks I did a "major maintenance" after 4 and a half years that it was operational (and without maintenance done previously because it was never necessary and there were never any problems) since it had reached less than 200GB unallocated. On it I have some acronis backup files "always incremental" which are basically cow and they have come to generate a big fragmentation and also a significant increase in space in the long run, and a big size also in metadata because of the huge fragmentation. After defragmentation and balancing the difference was remarkable.

EDIT:
I did a fast search and found this post that seems good and can be useful:

Post in thread 'Correction of ZFS write amplification'

Jun 8, 2023

I did a lot of testing over the years and never got the ZFS write amplification significantly down.
- try to avoid encryption if possible (doubles write amplification for whatever reason)
- try to avoid CoW on top of CoW
- try to avoid nested filesystems
- don't use consumer SSDs without PLP as these can't cache sync writes so the SSDs can't optimize the writes for less wear
- a raidz1/2/3 isn't great as a VM storage (less IOPS and problem with padding overhead) but total write amplification will be lower, as not everythign will have to be written twice (5 disk raidz1 will only write those...

• Dunuin

• 

 

[alexskysilk]

btrfs can be mounted using the nocow flag, which will mitigate SOME of the write amplification issues but not all. I would suggest you take a step back and try to define WHAT you want to accomplish without obsessing about HOW- would probably yield better options.

simplest thing is to either encrypt the underlying zfs file system, or use luks/lvm inside the guest and call it a day. btrfs offers no further utility here.

 

[Kingneutron]

> I currently have a ZFS pool of SSDs where I have some VMs on zvols (ext4 fs), and I want to create a VM with an encrypted disk that would be encrypted using LUKS and use BTRFS as a file system

Dude, you are so far out in the weeds that you can't see the sun anymore. That's literally one of the worst ideas that I have ever seen. Why not just use ZFS native encryption, or something like tomb?

 

[stckptr]

Thanks for your replies. To clarify my needs I want to be able to decrypt the VM disk from the guest itself via SSH (which I usually do with dropbear during the boot process) : if I encrypt the underlying zfs volume I won't be able to decrypt it from the guest.

As for btrfs it is not a hard requirement, I mostly wanted to use its compression capability since the files I manage are easily compressible. I just wanted to understand the limitations of doing this inside a zvol.