Proxmox with ZFS or BTRFS?

[N0Tallow3D]

Hey Guys,

new year, new challanges.
This year i picked filesystems and after 3 weeks reading documentation, tutorials and best practices I'm just running in a deadend.

I picked an existing power optimized server with 16 GB RAM, 1x 250GB nvme SSD (Proxmox+VM), 1x 256GB SSD SATA (Cache/Log), 2x 2TB HDD SATA (Raid1: Data), 3x 3TB HDD SATA (Raid1+Spare: Data).
The whole system isn't designed to be fast and the VMs contains only internal DNS, NAS and small services.

For my tests i made setups wie ZFS on both SSD and 2TB HDD as Raid1. The 3TB HDD are ext4. I'ved tested compress=off/lz4, dedup=on/off and serveral Cache/Log constellations on 256GB SSD.

In my second round I made setups with btrfs on the nvme SSD and luks+btrfs on 2TB HDD as RAID1.

Regardless what I'm doing the maximum transferrate (write) is between 40-60MB on ZFS. With btrfs I got 130-160MB. Is ZFS do much slower? I didn't find any errors and no tuneing tipps really work for me. So the conslusion should be btrfs espacially when there comes productive data. But one HHD randomly throws I/O errors. The Disk is ok (SMART&fsck&scrub), sata-cabel is changed (3 different types), controler is changed (PCIe and onboard) and src-data comes from 3TB HDD as the setup without any error with ZFS.

Actually I would prefer btrfs cause much more speed on same hardware but potential datalost isn't an option.

So.... What would you do? Did i missed a point?

Kind Regards
N0Tallow3D (to leave ext4 *g*)

 

[guletz]

So.... What would you do? Did i missed a point?

Yes you have miss a lot of points:

- btrfs is not integrated in the PMX web interface (for many good reasons )
- btrfs develop path is very slow with less developers compares with zfs (see yourself how many updates do you have in the last year for zfs and for btrfs)
- zfs is cross platform (linux, bsd, unix) but btrfs is only running on linux
- btrfs have a good reputation regarding how to lose your data (lvm corruption is only one of them)
- pool management for btrfs is only a bad joke compared with zfs (the same for snapshots)
- btrfs can use only mirror, raid10 (as I remember), no spare disks, no cache(like arc, l2arc for zfs), and NO replication

I have use both of them (around 9 years with zfs and I start to test btrfs 5 years ago). For many servers (tens or more with zfs and only 2 with btrfs) I never lose any data on zfs, but I have lose all my data on btrfs.

Now regarding about with yours speed tests(any tests), in my opinion (zfs/btrfs) the value is like /dev/null. I also try some tests and the results was very good. And in other cases was very bad. In a real world with real data/load I see many cases when my tests was not confirmed by real situation. So now I do not lose my time with tests. I use only real data with real load and do the best I can.
Maybe I am too stupid to use the proper tests

Good luck with btrfs, you need it

 

[alexskysilk]

Actually I would prefer btrfs cause much more speed on same hardware but potential datalost isn't an option.

If speed is desired, your whole config works against you regardless of file system choice. by separating your luns into multiple raid 1 volumes you're getting the worst of all worlds- least amount of usable data and only one disk's worth of command queue for any purpose. it will be slow regardless of file system choice. I wont belabor the point of btrfs vs zfs (@guletz did that already) but if I were you I'd make sure all my disks are the same and use them in a striped mirror configuration; you can then create extents for all your different functions and get . You wont get any actual benefit from your nvme or ssd drives the way you're using them; worse, on nvme failure you'll have a dead system.

 

[ricardoj]

Hi guys,

Some messages about BTRFS are outdated like 5 years or so ( experiences someone had in the past ).

I'm running 3 Open Media Vault and one Proxmox 5.4 ( Lab environment ) with BTRFS and I see no problems ( untill now ).

I decided to give BTRFS a try after a lot of Internet searches and found that a NAS manufacturer is selling boxes with it.

https://www.synology.com/en-us/dsm/Btrfs

I'm in no way affiliated with that manufacturer and / or with BTRFS development.

Also I'm not saying ZFS is no good or that BTRFS is the best option in the world.

From my personal experience ZFS needs an enterprise class hardware like fast disk and lots of memory / entreprise class SSD.

If you have the budget go for it but if you are low on budget and your hardware is not that fast it is worth try BTRFS

Please, see my other post about container backup on ZFS ( my environment ) :

https://forum.proxmox.com/threads/container-slow-backup.53499/

Best regards,

Ricardo Jorge

 

[guletz]

From my personal experience ZFS needs an enterprise class hardware like fast disk and lots of memory / entreprise class SSD.

And in my case I use(for at least 5 years) mostly non-enterprise hardware (no ecc ram, nas/surveillance sata hdd) with 8 GB ram (16 GB in the best case). And no ssd. But I must say that I spent unnumbered nights and neurons of my brain until I was able to start to understand zfs !

Even now I have some desktop class system.

So, different guys, different experiences. The most important think is that we can have different opinions(zfs, btrfs, nilfs2, whatever), and we can chose what is best for us/landscape.

 

[ricardoj]

HI,

I can't say that "different guys" is the only answer but different number of VMs as well as different VM OSes.

In my case with 18 Windows 10 workstations ( VDI ) running on "desktop class" hardware with 64 GB of RAM is a performance nightmare with ZFS.

Now talking about "choose" something it is not true as you can only install ZFS from the installation media.

Everything related to BTRFS must be done manually.

Yes you can do it but you have no support.

Regards,

Ricardo Jorge

 

[guletz]

Now talking about "choose" something it is not true as you can only install ZFS from the installation media.

True. But you can install a debian os(mirror) then pmx system on it. Then you can use btrfs as you like. All unsuported

Sorry for my bad english!

 

[Vladimir Bulgaru]

I can't same i'm a veteran in the field, but i've spent the past 6 months doing the best i can to protect the data on my server. As usual, the experience starts with understanding what is SAS, a RAID controller, RAID, redundancy, backup, data integrity and bla bla bla.

In short, i'd say that i find myself liking ZFS. The main con is the resource usage, but it gets better from there. Similar to btrfs it does protect data integrity by creating checksums and logging the writes. At the same time, ZFS is built into Proxmox and can be seamlessly deployed during installation. I don't like the fact that there is no GUI to tune ZFS settings and things need to be handled via the terminal, but i guess this can be said about a ton of things with Proxmox.

I also enjoy the fact that ZFS provides a great alternative to RAID controllers (was happy to get rid of this point of failure) and provides not only data redundancy, but also data integrity. As for resources, having 10-20GB consumed by ZFS and maybe adding an SSD for caching seems like a great deal when it comes to sleeping well while knowing your data is safe. Moreover, the cost for extra RAM is offset by the savings of not getting an expensive HW RAID controller.

As for btrfs, i read quite some articles on the topic and given how much ZFS has grown over the years, i do not see clear advantages. Here is the video that i can share on the topic, although it sounds outdated when it come to ZFS: https://www.youtube.com/watch?v=yAuEgepZG_8

 

[ricardoj]

Hi,

As I said one can install BTRFS by hand but have no support and the main reason in my opinion is the bad reputation because people still write "I have problems so many years ago when running BTRFS".

If BTRFS is so bad why SuSE is using it ?

Why a NAS manufacturer is selling boxes with it ?

Why Oracle supports it ?

Why Open Media Vault is planning to turn BTRFS as the default FS ?

If people only read about problems no one is going to think about BTRFS as an option and ZFS is the only "option" left.

There is no perfect FS and different systems may need different FS.

ZFS uses more resources than BTRFS and if you need a budget system why not use BTRFS and have it supported by Promox.

If you are going to use only enterprise class hardware may be you're OK with ZFS.

Regards,

Ricardo Jorge

 

[alexskysilk]

ZFS uses more resources than BTRFS and if you need a budget system why not use BTRFS and have it supported by Promox.

There are two issues with your supposition.
the first- BTRFS isnt really supported for any real "production" application. Sure, its fairly stable and has nice features but since there isnt any real anchor customer for it there isnt any major developer pushing the code. ZFS is used by a number of commercial applications including but not limited to LLNL Lustre, Datto, Nexenta, etc who contribute continuing developer time and resources. With redhat effectively announcing they have no further interest, SUSE is now the sole "torchbearer" for btrfs but they dont have any product tied to it (yes, there's synology but I wouldnt consider their engineering capability on par with the above. Incidentally, even they only use btrfs as an LVM, not as a raid substitute) If you're going to design a product that depends on effectively orphaned code, you better be able to support it yourself. File systems are massively complex and new edge case problems manifest all the time- just because you havent gotten bit yet doesnt mean it wont happen to you; how sure are you how long (if at all) it will take for the problems to be resolved?

the second- Just because you CAN doesnt mean you should. The developers of the product you actually want (Proxmox) have no interest in taking on the massive task of supporting another file system by themselves, and consequently all the relevant features are not built in to their product to support your choice. Is it really worth all the hoops jumped and wheel reinvention?

If you really dont want to use ZFS due to performance constraints, LVM atop MDraid works just fine.

 

[ricardoj]

Hi,

I'm not a of BTRFS fan boy and I hope all comments here do the same regarding ZFS !

BTRFS is still on the Proxmox roadmap.

Maybe it's a mistake or a strategy for the future. Who knows ?

- https://docs.docker.com/storage/storagedriver/btrfs-driver/
- https://www.synology.com/en-us/dsm/Btrfs
- https://docs.oracle.com/cd/E37670_01/E37355/html/ol_about_btrfs.html

The biggest problem with ZFS, from a Linux user's perspective, is the licensing. ZFS was licensed CDDL, which is a semi-permissive license that conflicts with the GPL. There is a lot of controversy over the implications of using ZFS with the Linux kernel, with opinions ranging from "it's a GPL violation" to "it's a CDDL violation" to "it's perfectly fine, it just hasn't been tested in court."

Regards,

Ricardo Jorge

 

[alexskysilk]

I'm not a of BTRFS fan boy and I hope all comments here do the same regarding ZFS !

The biggest problem with ZFS, from a Linux user's perspective, is the licensing.

You may claim to not be a fanboy, but when you switch over to describe reasons why ZFS is not for you instead of arguing why btrfs IS it becomes somewhat obvious. I do wonder, where did you get this idea that zfs licensing is a problem for a linux user at ALL, much less the BIGGEST? you may want to look here: https://github.com/zfsonlinux/zfs/wiki/FAQ#licensing

I will boil it down to make it easy:
1. there is NO provision in CDDL that prevents its use by Linux or any other OS
2. there is NO provision in GPL that prevents the inclusion of binary code including KLM (kernel loadable modules.) The limitation is on including it in the kernel, which no one is doing.

I really wish people would stop spreading that particular nonsense.

 

[ricardoj]

Hi,

As I said "BTRFS is still on the Proxmox roadmap."

If Proxmox guys see no value in BTRFS, they can change their roadmap.

It is up to them !

Talking about license :

- https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

Now about spreading nonsense and FUD that is why I revisited this thread.

Looks like the main reason to use ZFS as the only supported FS is "I had a problem many years ago when running BTRFS".

Why not double check and see if this is still true today when many other developers are running BTRFS ( please, see the links ).

There is no need to be scared about BTRFS.

Regards,

Ricardo Jorge

 

[praxmaxl]

From my personal experience ZFS needs an enterprise class hardware like fast disk and lots of memory / entreprise class SSD.

actually ECC RAM seems to be essential for ZFS, as per this article

 

[guletz]

Looks like the main reason to use ZFS as the only supported FS is "I had a problem many years ago when running BTRFS".

Hi,

IMHO, many users especially from enterprise zone, are not even think that BTRF could be a long term option for some good reason:

- RHEL for exemple do not suport BTRFS
- development is slow(including bug patching), and in some periods I saw only 2 updates in a year
- RAID 5, 6 is implemented but not ready/recomandable for production(as I remember)

And speaking from myself(but are many others like me for sure), I do not use anymore BTRFS not because I "had a problem many years ago" (problems could be solved) but because I lost ALL my DATA on btrfs. And I do not want to risk again and play at BTRFS lottery !

But some others persons may have a different exeprience/IT landscape/risk assessment and they consider that BTRFS is OK. My best wishes for them!


Good luck / Bafta!

​

 

[guletz]

actually ECC RAM seems to be essential for ZFS

Hi,

Almost true. But even now I have several old systems(with zfs for more then 7 years ) with no-ECC RAM, and I do not had have problems. One
of them is so old like this(zfs-fuse), see the date from zpool history:

2012-01-23.09:11:20 zpool create myzpool /dev/md6

... with HDD power-on-hours:

 9 Power_On_Hours          0x0012   084   084   000    Old_age   Always       -       114009

Good luck / Bafta !

 

[Helmut101]

Now that BTRFS is supported by Proxmox VE 7.0, how would you evaluate the choice for ZFS <-> BTRFS?

I currently have:

• 2x 16TB HDD Raid 1 (Proxmox storage)
• 2x 256GB SDD Raid 1 (for Hypervisor itself + VM data/temp)
• 2x 16GB HDD Raid 1 offsite

I want to switch to either ZFS or BTRFS to avoid silent data corruption. Offsite, I will continue to use rsync/borg//Raid1 Hardware Raid because it is low-level-robust and what I am familiar with, so there is not a crazy need for snapshot backups.

My system:

• 36GB ECC ram, 10 GB in use
• Single node, home use
• Primarily Unprivileged LXC containers, with Bind Mounts directly from host system/with user mapping
• A lot of nested Docker containers in unprivileged LXC, mainly for convenience
• USV double-conversion
• Offsite backup system: No USV

I need (ordered):

• Reliability. But I do have offsite backups, so a loss of data on one site would not be life threatening.
• Flexibility. I want to plug-in new drives and somehow extend my current storage; or replace single-drives without headache
• Encryption. I currently use Luks/dmcrypt, which is mounted on boot. I know that ZFS has native encryption support.
• Docker. I want to continue to use Docker - I have heard that the r/w performance of Docker inside LXC may be significantly reduced with ZFS as the underlying fs, with some rather complicated workarounds available (officially, docker supports both BTRFS and ZFS filesystems).

ZFS caveats:

• poor performance with Docker-LXC nesting
• requires more resources; requires ECC ram (no issue for me)
• ZFS Docker Storage Driver only supported for Ubuntu (I run all my Docker in Debian LXCs)

BTRFS caveats:

• integration is currently a technology preview in Proxmox VE
• no native file encryption, must be combined with luks/dmcrypt

I would really appreciate any feedback from those who have experience with either one of BTRFS/ZFS, or both filesystems.

 

[praxmaxl]

I want to switch to either ZFS or BTRFS to avoid silent data corruption. Offsite, I will continue to use rsync/borg//Raid1 Hardware Raid because it is low-level-robust and what I am familiar with, so there is not a crazy need for snapshot backups.

as you seem to have ECC RAM in place I guess I would follow the advise from here which basically says ... statistically no dataloss whatsoever reported by ZFS-users, dataloss reported on various occasions by btrfs-users.

And that is not my opinion but the advice of the head of the Linuxcontainers Team @Canonical. I personally would be am leaning towards BTRFS even as it is native on linux. That being said ... if I had ECC I'd probably go for ZFS based on that statement.

 

[Helmut101]

Interesting! Many thanks. I wonder why the quoted BTRFS docs suggest that the ZFS Storage driver is not usable inside a container, when the Docker docs actually have a separate section for using the ZFS storage driver, and LXC containers appear to be fully compatible with ZFS.

 

[praxmaxl]

I wonder why the quoted BTRFS docs suggest that the ZFS Storage driver is not usable inside a container, when the Docker docs actually have a separate section for using the ZFS storage driver ....

presumably you don't want to regard Linux Containers (LXC/LXD) == docker concept wise (eventhough both may be standing on similar grounds (cgroups, namespace). I can't really elaborate in that with sufficient background knowledge but would say it's like comparing apple's with pears.