Is there any way that iSCSI can offer snapshots & HA

[Tmanok]

Hi Everyone,

We've been struggling moving away from a mal-performing CEPH cluster at a customer site for a little while now, however, our NAS supports iSCSI and NFS, but not CEPH or ZFS over iSCSI. Is there truly no solution to include HA, Live Migration, and PBS Live Backups using an iSCSI NAS? We've considered numerous options, but this seems like a huge glaring limitation that VMWare, Hyper-V, and Xen have all solved but Proxmox is still behind on (due to it lacking locking mechanisms or locking-aware storage solution on iSCSI I take it).

Edit: Yes, we understand that NFS will permit .qcow2 which will enable snapshots for VMs but we are hoping for iSCSI performance and LXC snapshots too.

Please let me know if we're missing something obvious.
Thank you,


Tmanok

 

[bbgeek17]

Is there truly no solution to include HA, Live Migration, and PBS Live Backups using an iSCSI NAS?

You can absolutely use Proxmox with its standard iSCSI support and get HA, Live Migration and PBS backups (which dont use underlying storage mechanism and rely on Qemu).

We've considered numerous options, but this seems like a huge glaring limitation that VMWare, Hyper-V, and Xen

All commercial products with proprietary technologies addressing mostly a different segment of the market at quite a different cost from PVE.

have all solved but Proxmox is still behind on (due to it lacking locking mechanisms or locking-aware storage solution on iSCSI I take it).

iSCSI is just a protocol. Same limitations would apply to FC, NVMeOF, shared SAS, etc.

There are 3 categories of solutions (storage types) :
a) fully supported by PVE listed here https://pve.proxmox.com/wiki/Storage . As you discovered none provides an equivalent of VMFS or CFS. You said you have NFS available, doesnt that solve most of your issues? Granted, NFS, as implemented by many 3rd tier vendors, leaves a lot to be desired. It can however work in many situations.
b) Commercially supported storage solutions, like Blockbridge.
c) DYI solutions that you have to build and support yourself, ie OCFS2/GFS.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Tmanok]

You can absolutely use Proxmox with its standard iSCSI support and get HA, Live Migration and PBS backups (which dont use underlying storage mechanism and rely on Qemu).

Pretty please, describe the solution to me then using iSCSI without ZFS over it. NVM you mentioned OCFS2 and GFS2, but if there are any already employed by PVE please let me know.

iSCSI is just a protocol. Same limitations would apply to FC, NVMeOF, shared SAS, etc.

Yes, that's entirely true but I did not mention those as they are not within the scope of our available options. Mind you, SAS being treated as a local block device can be used for local ZFS, same is true for e-SAS.

DYI solutions that you have to build and support yourself, ie OCFS2/GFS.

I did actually set this up on a RedHat server late last year, fortunately there was product guidance from a commercial vendor at the time. If this was my homelab I'd consider it, but until Proxmox supports it, I'm not risking my client's butt for the sake of iSCSI over NFS.

Thanks bbgeek!

Tmanok

 

[bbgeek17]

Pretty please, describe the solution to me then using iSCSI without ZFS over it.

There are two built-in/native iscsi storage providers in PVE that dont involve ZFS:

root@pve7demo1:~# pvesm help add
USAGE: pvesm add <type> <storage> [OPTIONS]

  Create a new storage.

  <type>     <blockbridge | btrfs | cephfs | cifs | dir | glusterfs | iscsi
             | iscsidirect | lvm | lvmthin | nfs | pbs | rbd | zfs |
             zfspool>

Obviously they dont come with Proxmox native volume management or snapshots. Thin provisioning and HA would depend on your backend storage.

I did actually set this up on a RedHat server late last year, fortunately there was product guidance from a commercial vendor at the time. If this was my homelab I'd consider it, but until Proxmox supports it, I'm not risking my client's butt for the sake of iSCSI over NFS.

I may be wrong, but I dont believe that support for clustered filesystems is on PVE roadmap. Feel free to reach out to them and ask.

As far as I know there is only one solution for Proxmox that provides full PVE storage API integration with iSCSI protocol - Blockbridge.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

I may be wrong, but I dont believe that support for clustered filesystems is on PVE roadmap.

You have always to possibility to use GFS or OCFS2, which are (dedicated) cluster filesystems.... it's Linux so you can do what you want.

 

[bbgeek17]

You have always to possibility to use GFS or OCFS2, which are (dedicated) cluster filesystems.... it's Linux so you can do what you want.

Of course, see my comment 2 option (c). It seems that OP is looking for a vendor supported solution

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

Of course, see my comment 2 option (c). It seems that OP is looking for a vendor supported solution

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Yes, I have and just wanted to say that there IS a working solution. Best is to use a backend storage device which has integration into PVE if you want to have all those fancy features, like your blockbridge storage.

I just went with the ZFS-over-iSCSI route on real hardware and it works flawlessly.