iSCSI Live Migration Nimble

[Tmanok]

Hi Everyone,

One of my major clients is considering PVE but I've run into the same snag as earlier this year. The organization has several HPE Nimble SANs and they aren't going away as far as I can tell. HPE Nimble is block storage only, so iSCSI is basically the one and only solution available.

If we setup a four node cluster we need the ability to snapshot VMs for live migration and backup, but my understanding from some efforts earlier this year is that LVM lacks Metadata required to support PVE snapshots and LVM-Thin Metadata isn't able to be locked like with VMFS from VMWare or whatever Hyper-V Server uses (NTFS I presume).

Is there really no solution for this type of SAN which does not use ZFS underneath it? This is a critical question. Unfortunately the SANs don't offer NFS shares either and the client's team isn't willing to have another node in the middle to offer NFS from a single failure point.

Thank you for your time, if there is something in development at Proxmox to offer this feature it would immediately cause me to leap for joy.


Tmanok

 

[mira]

No, there's no way to get (thin) snapshots on LVM on top of iSCSI. They could use the LUNs from the storage directly and create snapshots on the storage side if it supports it, but there's no support for this on PVE.

 

[bbgeek17]

Is there really no solution for this type of SAN which does not use ZFS underneath it? This is a critical question. Unfortunately the SANs don't offer NFS shares either and the client's team isn't willing to have another node in the middle to offer NFS from a single failure point.

Before we implemented a fully integrated PVE storage Plugin for Blockbridge Storage we had a few customers who employed a manually managed model:
- Create Blockbridge virtual disks for each VM (boot and data)
- Attach the disks to all hosts in the cluster
- Configure Multipath
- Build a VM pointing to the attached disks
Even though the disks are seen by all hosts in the cluster, only one will access it any given time - where the VM is running. This is assured by PVE cluster management.
Snapshots and clones were handled directly on backend and invisible to PVE.

Obviously there is a lot of manual overhead involved and scripting the creation/removal/etc is critical. That said, it worked quiet successfully.

Since then we created and provide support to PVE API compatible storage plugin (https://www.blockbridge.com/proxmox/) where all the operations (create, delete, migrate, snapshot, clone, secure erase) are supported via PVE CLI/API/GUI.

My educated guess is that its unlikely that HPE will ever provide such integration.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Tmanok]

Before we implemented a fully integrated PVE storage Plugin for Blockbridge Storage we had a few customers who employed a manually managed model:
- Create Blockbridge virtual disks for each VM (boot and data)
- Attach the disks to all hosts in the cluster
- Configure Multipath
- Build a VM pointing to the attached disks
Even though the disks are seen by all hosts in the cluster, only one will access it any given time - where the VM is running. This is assured by PVE cluster management.
Snapshots and clones were handled directly on backend and invisible to PVE.

Obviously there is a lot of manual overhead involved and scripting the creation/removal/etc is critical. That said, it worked quiet successfully.

Since then we created and provide support to PVE API compatible storage plugin (https://www.blockbridge.com/proxmox/) where all the operations (create, delete, migrate, snapshot, clone, secure erase) are supported via PVE CLI/API/GUI.

My educated guess is that its unlikely that HPE will ever provide such integration.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi BBGeek,

Not only is that a lot of manual overhead, but if there is no integration with PVE, then there is no live migration or backup... This is not an acceptable solution unfortunately.

No, there's no way to get (thin) snapshots on LVM on top of iSCSI. They could use the LUNs from the storage directly and create snapshots on the storage side if it supports it, but there's no support for this on PVE.

Thank you for confirming my understanding, that is really too bad that PVE has not developed a solution for this like VMWare, Hyper-V, FlexVDI, and XCP-ng have. On a quick note about my time with FlexVDI (a CentOS Hypervisor), they used OCFS2.

There is already an enhancement request for PVE to have support for OCFS2, I've updated the request.

Thank you for your time,


Tmanok

 

[bbgeek17]

Not only is that a lot of manual overhead, but if there is no integration with PVE, then there is no live migration or backup... This is not an acceptable solution unfortunately.

Agreed, which is why we developed the PVE Plugin: https://kb.blockbridge.com/guide/proxmox/

On the other hand, if someone has a significant investment in an unsupported storage - manual, or OCFS route are still valid options.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[fabian]

or writing your own storage plugin

we only integrate open source storage stacks ourselves (and like with other features, only if it makes sense to us), but the storage plugin API is there for a reason, and there are a few third-party plugins (like Linbit's DRBD, blockbridge's, alternative freenas plugins, ..) already that leverage it. so if your nimble box has an API where you can sensibly manage snapshots and clones, you should be able to write a storage plugin that forwards these operations from PVE to the storage box.