iSCSI, snapshots and LXC within a PVE cluster

[Limoni dara]

Hello,

I’m currently installing 3 pve hosts in cluster mode with an iSCSI EMC array, everything is redundant : 2 ToR switches for the servers , 2 switches for the iscsi traffic. We’ve chosen this solutions to migrate from ESX (too expensive).
I’ve read a few articles / posts related to PVE clusters with an iSCSI SAN but doubts remain with the DELL EMC iSCSI array and the question of snapshots.

From what I understand :

It’s not possible to have lvm-thin (which would allow snapshots) with a shared storage.

So There’s 2 solutions after this :

1) The Open-iSCSI initiator (open-iscsi package) backend solution with which you can create lvm-thick VMs and LXC containers.
= NO SNAPSHOTS…

Or

2) The user mode iSCSI Backend which can be used with the package libiscsi2 to create VMs using iSCSI LUNs directly.
However, as stated in the ProxMox documentation, it comes with the drawback that you cannot use LVM on top of such iSCSI LUN so you need to manage all space allocations at the storage server side.
And also you can’t create LXC containers with this solution.

= SNAPSHOTS but directly on the array.​

I would need advice on these points:

1) If we chose the first solution (we were accustomed to snapshots with VMware I’m wondering if we will be able to do without easily ): is PBS a solution that can “replace” snapshot easily (in case of an OS update gone wrong, is it long to restore a VM?)
2) With the 2nd solution I have to make a LUN and then create a logical volume within the disk array management tool for each VM ? And only then I can attach these volumes within pve GUI and configure multipath? Is this right? has anyone tried to automate this (ansible or other tool) ?
3) We were not using LXC but we were using docker, will I miss the LXC option in case we chose the first solution?


Thanks for taking the time to read me

 

[bbgeek17]

It’s not possible to have lvm-thin (which would allow snapshots) with a shared storage.

Your understanding is correct.

1) The Open-iSCSI initiator (open-iscsi package) backend solution with which you can create lvm-thick VMs and LXC containers.
= NO SNAPSHOTS…

again, correct. No PVE supported snapshots. However, nothing stops you from using array snapshot capability. Array snapshot rollback will mean that every VM hosted on that LUN will be rolled back.

However, as stated in the ProxMox documentation, it comes with the drawback that you cannot use LVM on top of such iSCSI LUN so you need to manage all space allocations at the storage server side.

You can use LVM inside the VM and get snapshots, or you can use any filesystem that provides built-in snapshots, ie xfs. You will need to over-allocate space to account for snapshot growth.

: is PBS a solution that can “replace” snapshot easily (in case of an OS update gone wrong, is it long to restore a VM?

Backups and snapshots are solutions with different goals. Obviously a full restore of 1TB will take significantly longer than a snapshot rollback. On the other hand a 32GB root disk restore might be in acceptable range. A lot will depend on your PBS server spec and network. There is no definitive answer to your question.

With the 2nd solution I have to make a LUN and then create a logical volume within the disk array management tool for each VM ?

I think you got order of operations wrong. If you are creating a LUN per virtual disk, then you: create lun, attach to hosts, configure multipath, pass through DM device as raw disk to VM, do whatever you want inside the VM. Automation is always possible to the extent that the backend array allows. You'll need to study Dell/EMC CLI/API capabilities for your array, if any.

The other option is to pick a storage solution with native Proxmox integration

3) We were not using LXC but we were using docker, will I miss the LXC option in case we chose the first solution?

If you decide to use Docker, then you will likely run it inside a VM - it will be managed completely independently from Proxmox. Many people run docker/K8 without Proxmox, so only you can evaluate what if anything you will miss, or not.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Limoni dara]

Thanks a lot for your answer,


I think we will do as you detailed below and use the array snapshot capacity. + PBS (and stick to docker for containers)

I think you got order of operations wrong. If you are creating a LUN per virtual disk, then you: create lun, attach to hosts, configure multipath, pass through DM device as raw disk to VM, do whatever you want inside the VM. Automation is always possible to the extend that the backend array allows. You'll need to study Dell/EMC CLI/API capabilities for your array, if any.

in this case How much bigger do you think we need to make each lun compared to the VM size ?

 

[bbgeek17]

in this case How much bigger do you think we need to make each lun compared to the VM size ?

Are you asking for a case where you will use backend array snapshot on entire LUN? If you are, then you should talk to you storage vendor implementation specialist. Its possible that they dont keep snapshot in the LUN, but rather in their pool or whatever its called, from which the LUN is carved.
At the very least it will depend on the rate of change and lifetime of the snapshot.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Limoni dara]

Are you asking for a case where you will use backend array snapshot on entire LUN? If you are, then you should talk to you storage vendor implementation specialist. Its possible that they dont keep snapshot in the LUN, but rather in their pool or whatever its called, from which the LUN is carved.
At the very least it will depend on the rate of change and lifetime of the snapshot.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I will do that, thanks again!

 

[einhirn]

For people landing here via google:

Someone has written a script to create snapshots via LVM - still not "PVE-Supported" as in "integrated in the UI", but looks promising:

https://forum.proxmox.com/threads/lvm-snapshot-script-for-shared-storage-lvm-over-iscsi-only.150068/

Currently (September 2024) it's only distributed as a forum attachment which requires a login to download, but I already asked them to host it somewhere more accessible...