Shared Storage with FC-SAN

[valentin_ops]

Hello,

I'm trying to get the best storage possible using fiber-channel SAN.

With Multipath, I'm able to get the SAN appear as disk on my nodes, and I'm able to create a PV/VG on it.

So I have now a Shared storage based on LVM, on my SAN, available on all my nodes.

The only issue I get, is about Snapshots. LVM doesn't support snapshots.

We tried a ZFS Pool on top of the SAN, but because it appear as not shared on Proxmox side, it cause migration issues (it's long like local storage).

Any idea how to get a Shared and Snapshot ready storage on top of Proxmox, based on FC-SAN ?

Thank you

 

[LnxBil]

Any idea how to get a Shared and Snapshot ready storage on top of Proxmox, based on FC-SAN ?

It's unfortunately impossible with LVM and there is no other supported solution available.

Other threads about this:
https://forum.proxmox.com/threads/fibre-channel-san-with-live-snapshot.41205/
https://forum.proxmox.com/threads/proxmox-cluster-with-san-storage.40756/
https://forum.proxmox.com/threads/fibre-channel-shared-storage-how.37986/

I presented a solution that we use in this thread. It's not pretty, but it works:

Create a HA-VM on top of LVM with ZFS inside, export it's datasets to the same cluster as a storage as ZFS-over-iSCSI and you'll have a snapshottable shared storage in your system. We use this for tests if we need to:

• host VMs on LVM
• move VM live to other storage with snapshot capability
• snapshot VM, do what we need to do with the VM
• move it back to LVM

This works for us and we also use PBS, so we also have very fast backup and restore if we need to.

 

[bbgeek17]

@LnxBil covered 99% of the situation. You do have, with most SANs, an option to do a LUN snapshot. This is not as flexible, obviously. Can't be driven by built-in PVE toolset, but, in a pinch, it can be used for general data protection.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Daniel Lauck]

Hello.
I´m facing the same question. We have a five nodes Proxmox Cluster, and considering adopt a central storage. We have two options from two vendors: first uses a Zadara storage with iSCSI, and the second requires the instalation of HBA hardware in each of my hosts, and then create a FC based storage.
@LnxBil, regarding the use of FC, you presented a solution that consists of using ZFS-over-iSCSI and a local VM. Could you share how to configure this VM to share its contents with the Proxmox using ZFS-over-iSCSI and the final cojnfiguration in the Proxmox side? I will question the storage vendors if any of the equipament supports this ZFS-over-iSCSI protocol.

 

[bbgeek17]

first uses a Zadara storage with iSCSI, and the second requires the instalation of HBA hardware in each of my hosts, and then create a FC based storage.

Neither of these particular options will give you snapshot support. Thin provisioning might be somewhat possible with Zadara, but not in a way that can be controlled by Proxmox.

regarding the use of FC, you presented a solution that consists of using ZFS-over-iSCSI and a local VM. Could you share how to configure this VM to share its contents with the Proxmox using ZFS-over-iSCSI and the final cojnfiguration in the Proxmox sid

You simply need to present the VM with raw LUNs from either FC or iSCSI. This could be a direct pass-through, or via hypervisor.

I will question the storage vendors if any of the equipament supports this ZFS-over-iSCSI protocol.

I dont believe Zadara uses ZFS , so you wont be able to use ZFS/iSCSI directly. This is why you would be deploying that intermediate VM - it will provide that missing ZFS layer, in addition to ssh driven iSCSI management.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

In addition to @bbgeek17 good answer, I do not recommend to run the ZFS-in-between-storage in a production environment with FC-HA storage. You will introduce a single point of failure (SPOF) there. If you know about it and have failsafe in place, it is a possible way to go.

 

[Daniel Lauck]

Thanks a lot for your replies! I will use iSCSI, that is stable and well documented.

 

[bbgeek17]

Thanks a lot for your replies! I will use iSCSI, that is stable and well documented.

Sure, if it works better for you - go for it. At the end of the day they both provide block level storage and have identical limitations as far as PVE is concerned.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox