Recommended configuration for a 3 nodes cluster with 2 nodes connected to a DAS ?

[iouej]

I am aiming to get a 3 nodes proxmox cluster with 2 nodes connected to a DAS with this topology :

The idea behind this configuration is to have an HA cluster and a HA storage. What I had in mind was to have a zfs pool accessible in an active/passive configuration between nodes and active/active betwen links of a node.
This way, only one node writes to the DAS maximizing the use of the two avaible links and if it fails, the other node takes over using its two links in the same way as the first node.

I have read about dm-multipath from my understanding this solution works for a single node connected to a storage array as you can see in the red hat manual : https://access.redhat.com/documenta...se_linux/7/html/dm_multipath/mpio_description
And I can't find examples with dm-multipath with two nodes attached to the same storage array. Neither, I can't seem to deduce from the examples I have seen a configuration to run a two-node setup.

During my research, I have came across this post https://github.com/ewwhite/zfs-ha/wiki which matches my goal plus one cluster node not connected to the DAS. Howewer, I can't wrap my head around on how to do this in Proxmox. Could you point me the right direction ?

Thank you for your help !

 

[iouej]

Is my idea or question that bad ?

 

[mac.linux.free]

you could not use zfs in this case...only lvm is supported in ha mode in your case.

 

[iouej]

Thank you for your answer! Could you elaborate on why lvm is the only option ?
Is that related to the "Shared" option for storage https://pve.proxmox.com/wiki/Storage ? My understanding of this option is that the "Shared" caracteristic is necessary if two nodes writes concurrently on the same volume. In my case, the nodes will never write on the same volume at the same time. Node 1 writes on it by default and Node 2 takes over in a failover scenario like in the last link of my first post.

 

[mac.linux.free]

Ok so I'm using Proxmox since the Version 1.1 and in the early days I used the same scheme you plotted. So a "real" shared storage can only be in cluster mode by using LVM because the SCSI IDs in Linux allow the same storage by managing it via the LVM. So for example, you define one shared VG and one VM can run on one host and another VM runs on host2. And live Migration is much faster. Nowadays, I never use the "real" shared storage anymore. ZFS Replication is much more versatile and eliminates the SPOF.

 

[bbgeek17]

The reason that LVM is supported as shared storage is because in Thick mode all of space/blocks are pre-allocated. The size of each LVM slice is thick and there is no danger of two nodes writing to same block even though the LVM daemon on each node is not aware of the other. The nodes see the data about slices from the disk.
Note that snapshots are not supported because the blocks needed for saving data are allocated on-demand and now you do have corruption possibility.

ZFS is similar in this sense to LVM Thin - blocks are allocated as needed by each ZFS daemon on each node. There is a very real potential for collision. I'd say "not if but when".

There are extensions and workarounds - they are not natively supported in PVE distribution.

In theory if you are extremely careful and ensure that one and only one node is active and servicing VMs at a time, you could rely on "hope and a prayer" that the other node never attempts to write to the storage inadvertently.

As to your question about multipath : Ignoring PVE and ZFS for the moment. Assume you have a shared DAS used in a cluster-aware setup. Make sure you see raw disks via multiple paths on both nodes. Configure multipath identically on each node. Place a cluster aware file system on top and you are all set.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox