Any way to leverage ZFS clustering for HA

[FuriousGeorge]

I realize ZFS is not a true clustering fs, in that it doesn't support locks, but that's also true of lvm-based ha as implemented by proxmox.

As ZFS doies support active-active failover/takeover, I'm wondering if integration is in the works or currently possible in a way I'm not seeing.

Thanks for reading.

 

[guletz]

Hi,

zfs was not designed for a clusterfs. It is a storage volume manager. But you can use any clusterfs on top of zfs. As a simple example you can use glusterfs(even in nfs mode if you want) on top on zfs for each proxmox node.

 

[FuriousGeorge]

Hi,

zfs was not designed for a clusterfs. It is a storage volume manager. .

Thanks for the response. I should have phrased my question more clearly.

ZFS has a "clustering" feature whereby volumes can be replicated and failed over to other nodes, creating an active-active "cluster".

https://docs.oracle.com/cd/E37831_01/html/E52872/godgc.html

(I'm not sure if this can be done with Zo, will attempt later.)

The difference between this and PM HA, if I'm understanding correctly, is that it only works once, and must be manually failled back to the original node. Presumably it uses zfs replication and changes the direction of it as needed.

I use PM in my office/lab and I pass GPUs to the VMs (unfortunately PM makes a bad choice for this). My goal was to set up replication and fast offline migration of my VM from one node to another.

PM HA would work, if I manually detach and reattach local devices, which I'm fine with, but it requires either remote shared storage (NAS is slow for desktop use) or replication with e.g. DRBD (which I hear terrible things about).

This is intended to be educational and a labor of love more so than functional, and there are no plans for production use, so hackish solutions are fine.

What I'll probably do is start out with pbe-zsync, a vm on each node, and some ugly implementation. However, this would be active-passive with offline migration, and I'm not sure how long it will take to sync back after modifying the 2nd node.

 

[guletz]

Hi again,

It is something that could be done with zfs. You can have let say a dual-link storage, with one link in a server A(active) and another link on server B(passive). The pool it can be imported by only one server at a time(see zpool multihost property)
). If the A is going down then the B can import and use the same pool(in a automatically mode if I remember - it was a discussion on zfsonline discussion list about this feature).
Another ideea is like you say pve-zsync. You could replicate your VMs from node A to the node B(let say one time in 5 minutes). In case of server A is broken, you can easy move your *.conf of each VM from node A to the node B (mv /etc/pve/nodes/A/qemu-server/*.conf /etc/pve/nodes/B/qemu-server). Then you could start all this VM on node B(and to be sure that nothing bad wil be happen, you need to disconnect any network cable on server A, so replication will be stopped from A -> B).I test this scenario(by hand) and in my case one VM(about 1 Tb data size) can start on the other PMX node in less then 60 sec. This can be done automaitically with various tools ... but I will do this with monit if you ask me

 

[FuriousGeorge]

Thanks! Appreciate the suggestion. I'm looking into it now.