Filesystem Features XFS vs BTRFS vs ZFS

[floh8]

For better decisions according filesystem choice I thought it would be a good thing to see the possibilities.

filesystem	XFS​	BTRFS​	ZFS​
Compression	with VDO layer	inline / offline	inline
Deduplication	offline	inline / offline	inline
Metadata extern	yes	no	yes, special device
Read/Write cache	with LVO or Bcache layer	with LVO or Bcache layer	L2arc / slog
Bit Rotate protection	no	yes	yes
Raid types	no	Raid 1+10	Raid 1+raidz(5)+raidz2(6)+ raidz3(7)+10+50+60...
Raid Rebalance	-	yes	no
disable deduplication feature in the file system possible	yes	yes	no
snapshot for VM and CT	only VM with qcow2	yes	yes

 

[LnxBil]

Is there a question?

a few notes:

• In ZFS, you can disable deduplication. This will however only apply to newly written blocks and the on-disk- DEDUP-table will still be there and always be part of the ZFS pool.
• bcache is possible with EVERY filesystem, but may be good or bad with respect to data integrity depending on the settings.
• snapshots are also missing. ZFS has a dataset (or pool) wise snapshots, this has to be done with XFS on a per filesystem level, which is not as fine-grained as with ZFS. You also have full ZFS integration in PVE, so that you can use native snapshots with ZFS, but not with XFS.

 

[floh8]

Is there a question?

no

In ZFS, you can disable deduplication. This will however only apply to newly written blocks and the on-disk- DEDUP-table will still be there and always be part of the ZFS pool.

yes, i know, but i mean to deactivate the functionality of the filesystem itself like disable cow

bcache is possible with EVERY filesystem, but may be good or bad with respect to data integrity depending on the settings.

Also in things like live resizing is it not practical

snapshots are also missing. ZFS has a dataset (or pool) wise snapshots, this has to be done with XFS on a per filesystem level,

Good advice, i will add it

 

[LnxBil]

Good advice, i will add it

You do not have VM snapshot capability for XFS, you have filesystem snapshot capability to snapshot EVERYTHING, not only and especially not limited to a VM.

 

[floh8]

You do not have VM snapshot capability for XFS, you have filesystem snapshot capability to snapshot EVERYTHING, not only and especially not limited to a VM.

You have snapshot capability if you use qcow2-files. Snapshot the hole filesystem is in practice not useful. I'm going to add this information in the table.

 

[LnxBil]

You have snapshot capability if you use qcow2-files. Snapshot the hole filesystem is in practice not useful. I'm going to add this information in the table.

Yes sure, that is true for EVERY file based storage.

For XFS RAID, I'd add "external" (which is also possible for EVERY storage)

 

[Enlightend]

You can rebalance ZFS raid expansions simply send/receiving a snapshot of the pre-expansion data to the same pool and swapping it out afterwards. Ofcourse, this requires that you have sufficient space, but you can use external storage as an inbetween.

This is internal in ZFS with the tooling of ZFS, so while it takes steps over it happening with a simple command or entirely automatically, it's "internal with steps".

This can be done without touching the pre-expansion data until everything is done and over with, so you have all opportunities to test the data before swapping it out.

So rebalance should not be "no", but "internal with steps".


Pretty much same deal with "disable deduplication feature in the file system possible". You can perfectly disable dedup in ZFS, the downside is that already dedupped data remains dedupped.

But, by doing the same as with rebalancing, you can get rid of the dedupped versions of the data.

 

[LnxBil]

But, by doing the same as with rebalancing, you can get rid of the dedupped versions of the data.

Unfortunately not the dedup table on the pool and in the I/O path, this can only be "removed by a new pool".

 

[floh8]

This is internal in ZFS with the tooling of ZFS, so while it takes steps over it happening with a simple command or entirely automatically, it's "internal with steps".

I cant imagine what u describe. Please give an example!

Pretty much same deal with "disable deduplication feature in the file system possible". You can perfectly disable dedup in ZFS, the downside is that already dedupped data remains dedupped.

This point deals not with the ability to deactivate deduplication rather to deactivate the requirement of the feature.

 

[alexskysilk]

It may be worth nothing that VDO provides dedup facility as well, making LVM/EXT4 be roughly equivalent feature-wise with LVM/XFS. Has anyone actually used VDO compression/dedup for real? I dont actually have any experience with this.

Another "line item" would be of interest is for use with shared storage. ZFS and BTRFS are nonstarters afaik; anything atop lvm is fair game, but I'm wondering if anyone has tried using the thin provision features of VDO with this application.

 

[floh8]

It may be worth nothing that VDO provides dedup facility as well, making LVM/EXT4 be roughly equivalent feature-wise with LVM/XFS. Has anyone actually used VDO compression/dedup for real? I dont actually have any experience with this.

VDO only offers Dedup inline. In contrast, xfs do it offline, what is better for performance. At the moment, I play with a test enviroment to test VDO compression speed, but in virtual enviroments VDO seems to have its drawbacks.

 

[tigerblue77]

Hello, VDO is now coupled with LVM which support/permit/offers RAID functionnality

 

[tigerblue77]

VDO only offers Dedup inline. In contrast, xfs do it offline, what is better for performance. At the moment, I play with a test enviroment to test VDO compression speed, but in virtual enviroments VDO seems to have its drawbacks.

Using VDO on my LAB (R720XD with +10TB) and performance are terrible

 

[floh8]

I know there are much work to do for the vdo developer.