ZFS on SAN LUN(s)
- Thread starter alma21
- Start date
-
- Tags
- block storage san zfs
Well, https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_storage_types lists it explicitly as "ZFS (local)"!
ZFS wants to have direct access to the hardware of the devices. This has been discussed multiple times for "pass-through disks or a controller?".
Running it on a remote block device (through any protocol) would be... brave. Or simply
Note: just my personal idea of a complex topic.
I am not sure what that means. But one main feature of ZFS is to have some "self-healing" features, which require redundancy - which requires multiple devices per vdev.
Running ZFS on a obfuscated device (eg, RAID LUN) isnt dangerous per se; its no safer then any other filesystem on raid- which is to say you have no checksum correction facility at the filesystem level so you're paying the zfs tax for no gain. Moreover, write amplification tends to be larger in such scenarios so there are some capacity and performance losses.
Yes. that works. I would consider btrfs as an alternative as it has the same utility with lighter resource load, but pve support isnt as fully baked.
Okay, I must admit I do not know much about that scenario - I've never tried to build such a thing.
But I am fairly sure that there are more pitfalls than a new user (and me too) may expect to be present. That's why I wrote "... brave".
Not everything technically possible is recommended. But your-mileage-may-vary
the advanced idea is to have 2 (Storage) VMs in a HA pacemaker cluster running under Proxmox VE .... each of the VM has access to SAN LUNs which host one or more zpools which can failover to the other VM .... the zpools are then exposed via iscsi targets to the Proxmox VE host via the default ZFS over iscsi plugin
in before @UdoB
thats a bad idea. you can make it work but it would be brittle, unsupported and unsupportable.
Oh, sorry, I can't leave that sentence uncommented because of "similar features" ;-)
Comparing LVM with ZFS easy. The result is easy too: they only have "logical volume"-management in common. Everything else is different!
Specifically LVM does not offer: an integrated filesystem (w/ LVM it is always stacked on top), transparent compression, self-healing, cheap snapshots, guaranteed integrity, self-healing, replication, potentially integrated encryption (not used in PVE) and probably some feature more...
There are definitely multiple reasons to skip ZFS. But it should be an "informed decision" - knowing what ZFS gives is a prerequisite.
See also: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/
Last edited:
Not uncommon, but fraught.
I've seen effort to make a HA ZFS filer solution for AT LEAST 20 years. There are 3 commercial solutions I'm aware of (Truenas/Nexenta/Intelliflash) but all three have either migrated/migrating to a different method of HA (ceph in TrueNAS, Nexenta and Intelliflash to whatever DDN decided to do with them if not just flush down the toilet.)
In the open source, it remains a build your own because the solution IS so brittle. ZFS REALLY doesnt want to give up control in flight and so the usecase is fraught with edge cases. I know people HAVE built HA ZFS filers so it can be done, I just wouldnt do it if I needed to support it.
sorry, but I also can't leave this uncommented :
(imho LVM is the most underrated storage "virtualization" toolset in Linux)
LVM included:
transparent compression: dm-vdo (incl.dedup + thin provisioning + zero block detection)
guaranteed integrity: dm-integrity
self-healing: dm-integrity + lvm raid 1 (5/6)
cheap snapshots: lvm-thin
further:
dm-cache & writecache
create/move extents or complete LVs to dedicated PVs/disks
not direct LVM integrated but extendable:
sync replication: combine LVM with FC/iscsi/NVMeTCP and lvm raid1 and you get sync block replication
async replication/snapshot shipping: lvm thin send/recv : https://github.com/LINBIT/thin-send-recv (+ bash/cron)
no FS integrated ... can be also a feature .... add xfs/ext4 on top and it inherits all above layers/functions
combine all this and even more with HA cluster/pacemaker + iscsi targets and (dumb) SAN(s) and you get a SAN Volume Controller like behaviour .... add some kind of (currently not existing) "LVMthin over iscsi" storage plugin to Proxmox VE and there will be a fully integrated/feature rich Server & Storage virtualization platform which don't have to wait for specific Storage vendor plugins
I initially thought that zfs could be used instead LVM for such a kind of solution ... but with this discussion/answers it doesn't look so
Last edited:
this is true, but have you ever implemented it? its slow as molasses.
mdadm parity raid... brrr. THAT is brave. also, there is no "self healing" of any kind.
This has the same problems as a HA zfs filer. the only "upside" is that you dont have cache synchronization issues but that kills the performance benefit of arc in the first place.
well here lies the rub. this doesnt exist because PVE cant support any sort of thin provisioning on a shared LUN without regard of the underlying storage topology. You CAN use lvm thin on your filer but there is no way to make pve aware of it, and it DOES create problems with runaway trims/snapshots on the underlying service provider; and also, as my comment above mentions you really really dont want to depend on mdadm for the disk aggregation function, and you'd really want a multi controller aware raid controller to do this.
IT IS POSSIBLE, dont get me wrong- but you really need to understand what the operational constraints are. There is a reason anyone who does this for real buys storage appliances.
I did not perform a fio benchmark or similiar .... I assume vdo dedup is similiar slow as ZFS dedup... but with compression only I think it can be reasonable fast ... benchmarks: to be done
it's lvmraid ... but yes it uses mdraid under the hood .... but for selfhealing lvm raid1 is also sufficient:
this I don't understand .... imho a pacemaker HA-LVM + LIO iscsi target + VIP resource agent etc is a supported setup under RHEL (8-10) at least ... I think also SLES and Ubuntu Server will support this ....
the shared SAN LUNs should of course have their own raid protection from the corresponding SAN/controllers .... the VM pacemaker cluster will only create PVs/VGs and lvm thin pool(s) out of the shared LUNs and only 1 vm storage node owns 1 or more VGs (+ thinpool + iscsi target + VIP) at any time which can failover to the other node ... the LVMthin over iscsi plugin should then work similiar as the zfs over iscsi plugin .... if clustered or not should not matter .... the plugin connects always to the corresponding VIPs
maybe I have to provide a sample config/layout ... this was anyway my intention
Last edited: