Why expressly linking btrfs to zfs, and not other filesystem? Let me explain myself.
PERFORMANCE VERSUS HA
Always, there’s a trade-off between performance and high availability.
With ceph distributed storage you have HA, but there’s performance penalty, induced by network latency.
With local lvm thin you have high performance, near bear metal ext4, without high availability ( lvm cluster works also with network shared storage or lvm mirrored).
With zfs, you can the best have both worlds:
Zfs has a lot of features, but in this context, send/receive is where it shines.
Let me give you an example: https://ayufan.eu/projects/proxmox-ve-differential-backups/ is a smart patch for Promox, based on http://xdelta.org/
Unfortunately, doing a differential backup takes more time than the initial backup ! ( xdelta3 is slow scanning all the files) .
On the other hand, I have done a paranoia test, with a 100GB LXC ZFS container, having more than 700.000 files. Using send/receive pve zsync, doing an incremental backup of another 8000 files, takes max 5 sec! The magic is that ZFS compares incremental the last snapshots, not whole subvolume. See code relevant code from pve-zsync
if($source->{last_snap} && snapshot_exist($source , $dest, $param->{method})) {
push @$cmd, '-i', "$source->{all}\@$source->{last_snap}";
}
push @$cmd, '--', "$source->{all}\@$source->{new_snap}";
BTRFS
Btrfs is the only filesystem that has built in send/receive incremental replication, like ZFS.
It used to be unstable, but now it seems to be ok.
BTRFS it’s not the best candidate for virtualization
For example, it suffers when there are heavy write activities in the middle of an existing files, so probably it’s not the best candidate for virtualization (the virtual disks are updated in-place at each write). http://www.virtualtothecore.com/en/2016-btrfs-really-next-filesystem/
https://phocean.net/2016/03/20/a-journey-with-btrfs.html reports data corruption.
BTRFS Performance
Btrfs doesn’t shine for databases https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp
Performance is poor http://www.ilsistemista.net/index.p...ith-kvm-a-storage-performance-comparison.html . This test is not relevant, with old linux kernel.
Update : http://www.phoronix.com/scan.php?page=article&item=4fs-linux-4649&num=4
My latest sysbench and fio tests showed good results, using btrfs as directory storage in proxmox, comparing to zfs and ext4.
Docker bets on BTRFS
https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/
Btrfs has been long hailed as the future of Linux filesystems. With full support in the mainline Linux kernel, a stable on-disk-format, and active development with a focus on stability, this is now becoming more of a reality. As far as Docker on the Linux platform goes, many people see the btrfs storage driver as a potential long-term replacement for the devicemapper storage driver.
So if this ok for docker, I guess could be ok also for LXC on Proxmox.
Current state on Proxmox
Of course now you can use as directory on proxmox https://www.internalfx.com/how-to-use-a-btrfs-raid-on-your-proxmox-server/ , but you are missing all the proxmox integration benefits ( including spanshots).
ZFS on LINUX LICENSING OUTLINE
Short outline story on Wikipedia https://en.wikipedia.org/wiki/Commo...icense#CDDL.27d_ZFS_into_GPL.27d_Linux_kernel
-Ubuntu says “yes” https://insights.ubuntu.com/2016/02/18/zfs-licensing-and-linux/
- Software Freedom Conservancy says “no” https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/, even with possibility of seeking resolution from the Courts, giving the example of https://sfconservancy.org/copyleft-compliance/vmware-lawsuit-faq.html#why-lawsuit ( funny thing the first round is lost http://www.theregister.co.uk/2016/0...pl_breach_case_but_plaintiff_promises_appeal/ )
On the other hand, I guess they don’t really want to bring it in court https://sfconservancy.org/blog/2016/apr/11/fsf-zfs/ , it’s more like an ambitious license conflict : Richard M. Stallman says “ Oracle should lead the way, as the largest single copyright holder in the ZFS codebase, to either relicense ZFS, or, if they prefer, publish an updated CDDL that is compatible with the GPLv2 and GPLv3 (which would be another viable route to solve the same problem).”
What is more intereseting ( see https://en.wikipedia.org/wiki/OpenZFS#History ) that the initial port of ZFS on Linux was in 2008, the OpenZFS umbrella in 2013, but only in 2016, with release of Ubuntu 16.04 LTS includes CDDL-licensed ZFS on Linux, all the internet fuss started. Now, after one year, I am not aware of any public updates on this issue.
The above are internet links. On the other hand, my 2 cents is that Oracle really enjoys staying silent in this dispute: Allowing the open source community working, but backporting innovations and fixes form OpenZFS to his closed source ZFS distribution. On the other hand, refusing to relicense ZFS under GPL, keeps a confusion for ZFS on linux, boosting the marketing and sales for his enterprise ZFS based products.
CONCLUSIONS
This boring legal staff sounds like pure paranoia. ZFS is so good, that I take my chances and go for it. Fortunately, the Proxmox flexibility allows me to port the LXC containers to another storage, just in case.
On the other hand, what if ? The integration of a new storage backend is not trivial, so why not putting btrfs proxmox on roadmap, even with low priority ?
Finally, to quote FreeNAS : “Once you go ZFS, you will never want to go back.”
PERFORMANCE VERSUS HA
Always, there’s a trade-off between performance and high availability.
With ceph distributed storage you have HA, but there’s performance penalty, induced by network latency.
With local lvm thin you have high performance, near bear metal ext4, without high availability ( lvm cluster works also with network shared storage or lvm mirrored).
With zfs, you can the best have both worlds:
- High local performance ( with goodies like ARC Cache, ZiL)
- High availability, using near continuous replication send/receive incremental of local zfs filesystem, to remote server
Zfs has a lot of features, but in this context, send/receive is where it shines.
Let me give you an example: https://ayufan.eu/projects/proxmox-ve-differential-backups/ is a smart patch for Promox, based on http://xdelta.org/
Unfortunately, doing a differential backup takes more time than the initial backup ! ( xdelta3 is slow scanning all the files) .
On the other hand, I have done a paranoia test, with a 100GB LXC ZFS container, having more than 700.000 files. Using send/receive pve zsync, doing an incremental backup of another 8000 files, takes max 5 sec! The magic is that ZFS compares incremental the last snapshots, not whole subvolume. See code relevant code from pve-zsync
if($source->{last_snap} && snapshot_exist($source , $dest, $param->{method})) {
push @$cmd, '-i', "$source->{all}\@$source->{last_snap}";
}
push @$cmd, '--', "$source->{all}\@$source->{new_snap}";
BTRFS
Btrfs is the only filesystem that has built in send/receive incremental replication, like ZFS.
It used to be unstable, but now it seems to be ok.
BTRFS it’s not the best candidate for virtualization
For example, it suffers when there are heavy write activities in the middle of an existing files, so probably it’s not the best candidate for virtualization (the virtual disks are updated in-place at each write). http://www.virtualtothecore.com/en/2016-btrfs-really-next-filesystem/
https://phocean.net/2016/03/20/a-journey-with-btrfs.html reports data corruption.
BTRFS Performance
Btrfs doesn’t shine for databases https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp
Performance is poor http://www.ilsistemista.net/index.p...ith-kvm-a-storage-performance-comparison.html . This test is not relevant, with old linux kernel.
Update : http://www.phoronix.com/scan.php?page=article&item=4fs-linux-4649&num=4
My latest sysbench and fio tests showed good results, using btrfs as directory storage in proxmox, comparing to zfs and ext4.
Docker bets on BTRFS
https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/
Btrfs has been long hailed as the future of Linux filesystems. With full support in the mainline Linux kernel, a stable on-disk-format, and active development with a focus on stability, this is now becoming more of a reality. As far as Docker on the Linux platform goes, many people see the btrfs storage driver as a potential long-term replacement for the devicemapper storage driver.
So if this ok for docker, I guess could be ok also for LXC on Proxmox.
Current state on Proxmox
Of course now you can use as directory on proxmox https://www.internalfx.com/how-to-use-a-btrfs-raid-on-your-proxmox-server/ , but you are missing all the proxmox integration benefits ( including spanshots).
ZFS on LINUX LICENSING OUTLINE
Short outline story on Wikipedia https://en.wikipedia.org/wiki/Commo...icense#CDDL.27d_ZFS_into_GPL.27d_Linux_kernel
-Ubuntu says “yes” https://insights.ubuntu.com/2016/02/18/zfs-licensing-and-linux/
- Software Freedom Conservancy says “no” https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/, even with possibility of seeking resolution from the Courts, giving the example of https://sfconservancy.org/copyleft-compliance/vmware-lawsuit-faq.html#why-lawsuit ( funny thing the first round is lost http://www.theregister.co.uk/2016/0...pl_breach_case_but_plaintiff_promises_appeal/ )
On the other hand, I guess they don’t really want to bring it in court https://sfconservancy.org/blog/2016/apr/11/fsf-zfs/ , it’s more like an ambitious license conflict : Richard M. Stallman says “ Oracle should lead the way, as the largest single copyright holder in the ZFS codebase, to either relicense ZFS, or, if they prefer, publish an updated CDDL that is compatible with the GPLv2 and GPLv3 (which would be another viable route to solve the same problem).”
What is more intereseting ( see https://en.wikipedia.org/wiki/OpenZFS#History ) that the initial port of ZFS on Linux was in 2008, the OpenZFS umbrella in 2013, but only in 2016, with release of Ubuntu 16.04 LTS includes CDDL-licensed ZFS on Linux, all the internet fuss started. Now, after one year, I am not aware of any public updates on this issue.
The above are internet links. On the other hand, my 2 cents is that Oracle really enjoys staying silent in this dispute: Allowing the open source community working, but backporting innovations and fixes form OpenZFS to his closed source ZFS distribution. On the other hand, refusing to relicense ZFS under GPL, keeps a confusion for ZFS on linux, boosting the marketing and sales for his enterprise ZFS based products.
CONCLUSIONS
This boring legal staff sounds like pure paranoia. ZFS is so good, that I take my chances and go for it. Fortunately, the Proxmox flexibility allows me to port the LXC containers to another storage, just in case.
On the other hand, what if ? The integration of a new storage backend is not trivial, so why not putting btrfs proxmox on roadmap, even with low priority ?
Finally, to quote FreeNAS : “Once you go ZFS, you will never want to go back.”
Last edited:
