[SOLVED] Sharing a ZFS pool between nodes over SAN (fiber) for virtual disks

[IT Hero]

Dear Proxmox community

I would like to build a Proxmox infrastructure with several nodes connected to a SAN through Fiber.

I plan to store virtual disks on a ZFS pool which will then be shared through all nodes.

One SAN disk will be seen as local drive (using device mapper/multipath) and it will be used as vdev in my ZFS pool.

Proxmox will then create a new dataset for each virtual drive.

I know that this works with LVM, but is it a recipe for disaster in the case of a ZFS/zpool ?
will there be corruption if several machines believe they're the sole owner of the zpool ?

Thanks for any light on this

Leo

 

[proxminent]

Hello Leo! SAN through fiber, does it mean that you can see the LUNs in the host or does your storage support ISCSI?

 

[IT Hero]

Hello Leo! SAN through fiber, does it mean that you can see the LUNs in the host or does your storage support ISCSI?

Hi proxminent !

all my nodes see the LUNs (through 8Gb fiber, connected over a Fiber switch)
the backend storage itself can provide iSCSI but over 1Gb

 

[proxminent]

Ok. SAN Modell ?

I think, the best option is to use LVM on top of FC-based storage.
This gives you LVM an shared storage (block) and live migration.
disadvantage: No Snapshots

 

[IT Hero]

Ok. SAN Modell ?

I think, the best option is to use LVM on top of FC-based storage.
This gives you LVM an shared storage (block) and live migration.
disadvantage: No Snapshots

Yes, LVM would work, I used it on another setup.

This time I would like to use ZFS. It is working on iSCSI because the target expose the zvols. But can it work directly on a shared zpool.

This sounds more like a ZFS question actually.

So each Proxmox node would manage the zvols that are used by its VMs/Containers but would not touch the rest.

But is there some race condition if 2 nodes try to modify the zpool (pseudo)simultaneously and corruption occurs.

Do I need to fence stuff like in clusters ? or does Proxmox do it alone ?

 

[IT Hero]

So, for future reference, I answer my own question:

On its own, ZFS is NOT cluster-aware.

If you want to access a zpool from several machines at once, you have to build a cluster using such tools as corosync, pacemaker and fence your drives etc... (which is a whole other topic).

Btw, LVM is also NOT cluster-aware (https://tldp.org/HOWTO/LVM-HOWTO/sharinglvm1.html), but Proxmox does it magic with clustering and makes it work. To my knowledge, it does not the same with ZFS, so you have to do it yourself/use ZFS over iSCSI.

I suppose a mod can now close this