ZFS vs LVM for Proxmox OS disk?

[victorhooi]

Hi,

As of 2019 - what are the pros/cons of ZFS vs LVM for the Proxmox OS disk?

(This won't be for actual VM disks - which we are storing on Ceph - or even for ISOs - stored on CephFS - but simply for the actual Proxmox system itself).

Thanks,
Victor

 

[LnxBil]

Pros ZFS:
- the only supported software raid for PVE
- better snapshots than LVM (easy rollback on errors)
- replication for backup
- compression

Cons ZFS:
- slightly higher hardware requirements
- bigger software stack than LVM (ZFS includes besides volume managenent also filesystem and raid capability)
- ZFS still not that rock-solid as LVM has been for decades, but ZFS is getting there
- currently no TRIM (in upstream, but not in PVE ZFS yet)

 

[victorhooi]

Thanks for that - for our use case, the replication for backup seems interesting.

Does that provide an easy way to backup an entire Proxmox OS to another remote ZFS server?

However, on the con side - the lack of TRIM seems pretty bad. We will be installing on SSDs. How bad in the real world is the lack of TRIM?

Is there a tracking link or discussion for getting TRIM into Proxmox's ZFS?

 

[guletz]

However, on the con side - the lack of TRIM seems pretty bad. We will be installing on SSDs. How bad in the real world is the lack of TRIM?

Is there a tracking link or discussion for getting TRIM into Proxmox's ZFS?

I use zfs on ssd for many years without any fstrim (Intel DC or even consumer Adata ). Mostly in mirror zfs. But once or maybe 2 / year, I do this:

- I have a spare ssd, that I keep in a closet
- I a nice week-end I stop one server and replace with the spare ssd (who is already trimmed )
- start the server, resync the zfs mirror
- run a scrub ...
- trim the ssd that was on server and a smartctl test, then -> store in the closet(in a metal box)
- then repeat the same for all others ssd (round robin fashion)

So in the worst case, my ssd's will have a longer life

So you need only one extra ssd/cluster, not only for fstrim (ssd could also be broke and you do not want to find a new one in 5 minutes )

 

[udo]

Hi,

As of 2019 - what are the pros/cons of ZFS vs LVM for the Proxmox OS disk?

(This won't be for actual VM disks - which we are storing on Ceph - or even for ISOs - stored on CephFS - but simply for the actual Proxmox system itself).

Thanks,
Victor

Hi,
unfortunality the performance for zfs is not so good, like with lvm and an hw-raid-controller.

We had some trouble with mysql-write-latencies on zfs (SSDs), which are gone, after switching to lvm-raid-storage (same SSDs).

Pro for features, like replication for disaster recovery...

Udo

 

[Gilles]

After more than 2 years "living" with zfs, I think this system is great for dev & experts and has some "miraculous" features like instant snapshot.

BUT you have to be VERY carefull in everything you are doing... I had and still have so many bad suprises (around replication, snapshots and backup/restoration) and most of them are unrecoverables. You have to be an expert before using it (like Oracle VS MySql or VMW VS PM ).

I do not sleep as well as during LVM old times... I only keep it due to the fact that PM only support ZFS softraid unfortunatly.

By the way OVH/So You Start SSD hardware do rarely support Proxmox ZFS distribution... and you never know in advance !
(I suspect they don't like fstrim/fragmentation fast deterioration problems)

 

[beemino]

I do not sleep as well as during LVM old times... I only keep it due to the fact that PM only support ZFS softraid unfortunatly.

I love zfs for many use. ZFS snapshots are very efficient for journalised backup storage .

Hoever I prefer having My PVE's OS on a good old mdraid with ext4 fs.
To achieve this, I always install pve on top of a fresh debian with mdraid.

 

[LnxBil]

Hoever I prefer having My PVE's OS on a good old mdraid with ext4 fs.

I hope that in the near future we will have a good Grub ZFS integration with Snapshots. The code for it is already present in the ZFS packages for some years now, but the entries are not generated per default. With this, you can have snapshots of your root PVE dataset (in ZFS of course) and boot a snapshot if e.g. a kernel-update fails and you cannot boot up your machine.