Fiber Chanel and Shared Storage - Snapshot supported (HA enabled)

[berkaybulut]

I will soon be purchasing IBM FlashSystem Storage for my 5 node Proxmox cluster. There are questions I'm wondering.

1 - ) I will provide 2 16G connections with FC from Storage to each node. How am I going to describe this to Proxmox.
2 - ) I have to use HA mode active. Any problems with live streaming with FC Link?
3 - ) According to my research I should use LVM to use FC connections as shared storage. But this does not support snapshot storage. How can I run this?

Thanks in advance for your responses.

 

[bbgeek17]

1 - ) I will provide 2 16G connections with FC from Storage to each node. How am I going to describe this to Proxmox.

You will use Multipath. Although this article is geared towards iSCSI/Multipath, the concepts are the same https://pve.proxmox.com/wiki/ISCSI_Multipath. The vendor (IBM) has instructions on the appropriate setup of their storage system with Debian.

I have to use HA mode active. Any problems with live streaming with FC Link?

Do you mean PVE HA? If you do - then yes, live failover will work fine. I am not sure what "live streaming" is in the context of PVE/HA/Storage.

According to my research I should use LVM to use FC connections as shared storage. But this does not support snapshot storage.

You are correct, PVE supported option is non-thin/thick LVM. No snapshot support in such configuration.

How can I run this?

Best option - buy storage that is Proxmox aware/compatible/supported.
A DYI option is to self-configure Cluster-Aware Filesystem (ie. OCFS2) and use it as directory storage with QCOW images.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[berkaybulut]

You will use Multipath. Although this article is geared towards iSCSI/Multipath, the concepts are the same https://pve.proxmox.com/wiki/ISCSI_Multipath. The vendor (IBM) has instructions on the appropriate setup of their storage system with Debian.

Do you mean PVE HA? If you do - then yes, live failover will work fine. I am not sure what "live streaming" is in the context of PVE/HA/Storage.

You are correct, PVE supported option is non-thin/thick LVM. No snapshot support in such configuration.

Best option - buy storage that is Proxmox aware/compatible/supported.
A DYI option is to self-configure Cluster-Aware Filesystem (ie. OCFS2) and use it as directory storage with QCOW images.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Is there a way I would follow to bring snapshot support? I'm considering using LVM with iSCSI Multipath. Because I have to use HA.

 

[bbgeek17]

Is there a way I would follow to bring snapshot support? I'm considering using LVM with iSCSI Multipath. Because I have to use HA.

If the equation is : FlashSystem FC storage + Multipath + Multi Host access (shared) + HA + XXX = snapshots, then the only reasonable solution to XXX, at this point in time, is OCFS2 or similar.

Or you you can replace FlashSystem with Blockbridge, where Blockbridge+Multipath+Shared+HA+Snapshot+Clones+Thin=Profit

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

AFAIK iSCSI with ZFS-over-iSCSI has also snapshot support but lacks the multipathing.

 

[berkaybulut]

If the equation is : FlashSystem FC storage + Multipath + Multi Host access (shared) + HA + XXX = snapshots, then the only reasonable solution to XXX, at this point in time, is OCFS2 or similar.

Or you you can replace FlashSystem with Blockbridge, where Blockbridge+Multipath+Shared+HA+Snapshot+Clones+Thin=Profit

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

As Storage, we gave up on IBM for certain reasons. We now have the DELL Unity 600F. Do you have an equation for this?

 

[narrateourale]

AFAIK iSCSI with ZFS-over-iSCSI has also snapshot support but lacks the multipathing.

Isn't ZFS over iSCSI basically ZFS on the storage host where PVE manages the ZFS volumes and snapshots via SSH and exports them as iSCSI LUNs?

Not ZFS on top of iSCSI LUNs?

 

[bbgeek17]

As Storage, we gave up on IBM for certain reasons. We now have the DELL Unity 600F. Do you have an equation for this?

IBM=DELL

Isn't ZFS over iSCSI basically ZFS on the storage host where PVE manages the ZFS volumes and snapshots via SSH and exports them as iSCSI LUNs?

This is exactly correct. One would have to attach IBM/Dell/etc to a supported Linux host via iSCSI or FC, then partition/format the LUNs into ZFS pools, then ZFS/iSCSI PVE storage plugin will facilitate export of the the ZFS volumes carved out from the pool using iSCSI to PVE.

The HA portion of the ZFS and iSCSI is left to the implementer.

One may also pass-through the LUNs to a VM running in PVE cluster, so the HA will be somewhat offloaded to PVE. Its up to the admin to weigh benefits vs complexity.

IMHO, in any semblance of production, ZFS/iSCSI makes sense only with something like HA TrueNAS, where there is no an extra intermediary. There is an unofficial fork of the ZFS/iSCSI plugin on github that is adapted for this.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

One may also pass-through the LUNs to a VM running in PVE cluster, so the HA will be somewhat offloaded to PVE. Its up to the admin to weigh benefits vs complexity.

Yes, this is what we do.

IMHO, in any semblance of production, ZFS/iSCSI makes sense only with something like HA TrueNAS, where there is no an extra intermediary. There is an unofficial fork of the ZFS/iSCSI plugin on github that is adapted for this.

Unfortunately, there is only snapshot support in dedicated HA shared storage solutions like @bbgeek17's company offers and this pains me a lot. The "old enterprise way" with a centralized HA storage does unfortunately not work well with PVE (besides NFS of course).

 

[bbgeek17]

Unfortunately, there is only snapshot support in dedicated HA shared storage solutions like @bbgeek17's company offers and this pains me a lot. The "old enterprise way" with a centralized HA storage does unfortunately not work well with PVE (besides NFS of course).

A stable production ready support for any hypervisor takes time, effort and money. I can say that we made and continue to make a significant investment in supporting Proxmox via dedicated proper Plugin to ensure stable functionality.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

A stable production ready support for any hypervisor takes time, effort and money. I can say that we made and continue to make a significant investment in supporting Proxmox via dedicated proper Plugin to ensure stable functionality.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

That is very good to hear!