[SOLVED] Using Directory instead of LVM with ISCSI multipath

[Sarge]

Hi,
By installing proxmox 7.4 on two nodes+shared ISCSI storage, and setting up ISCSI multipatch according to that manual(with LVM), I had a problem.
I can't take a snapshot of the virtual machine!
OK, I uninstalled LVM, vgremove, pvremove, and found an easier solution.

Formatted ISCSI multipatch:

mkfs.ext4 /dev/mapper/mpatha

Mounted this partition on each node:

mount /dev/mapper/mpatha /mnt/iscsi

And created directory in the GUI..

Thats worked! Snapshots of VM, are created in datastore! Disk format type, when creating a virtual machine, is also available
The big question, is whether I did the right thing, by removing one level of abstraction in the form of lvm?
Is my decision correct?

 

[bbgeek17]

The big question, is whether I did the right thing, by removing one level of abstraction in the form of lvm?
Is my decision correct?

A short answer - its incorrect.

A long answer - for two computers to be able to access the same disk, there needs to be something that will prevent them from stepping on each other and corrupting data. The "abstraction" you removed was the only thing standing in the way of data loss.
LVM thick is a layer that allows you to share the disk between multiple cluster members at a raw level in Proxmox. However it comes at a cost of snapshots. A Clustered File system is another method of achieving similar result, which will allow you to take snapshots.
These are the primary two ways in a combination of Proxmox+iSCSI that allow you to have shared physical disk. LVM thick is the only built-in one.

There are a few others more specialized solutions, such as "iSCSI over ZFS" and Blockbridge.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Sarge]

Thx, bbgeek17!
Well then, there remains only one question, which file system to choose for creating snapshots?
It seems to me that ZFS over ISCSI is too complicated solution.
What would you recommend in the case of shared storage?

​

 

[bbgeek17]

Well then, there remains only one question, which file system to choose for creating snapshots?

Any filesystem will support snapshots in Proxmox single node setup, if you use QCOW as disk format for disk image files. ZFS is another option for non-shared setups.

It seems to me that ZFS over ISCSI is too complicated solution.

The level of complexity depends on your skillset, patience and end-goals. It requires a specific combination of supported software. It will not work with 99% of commercial iSCSI storage.

What would you recommend in the case of shared storage?

As usual - "it depends". Is it a home setup or business production? If home - anything (ceph, clustered filesystems, iscsi/zfs, etc), try it.
If business, do you have skills to roll out and support DIY? Do you have budget to back it with appropriate hardware? Do you have defined performance and scale requirements? The answer will oscillate between Ceph and Clustered File Systems.
Do you want a commercially supported one stop solution with specific hardware BoMs and assisted deployment and 24x7 support? Blockbridge.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Sarge]

Bbgeek17 Thank you for the detailed answer.
I'll clarify:
This solution is designed for a business project with 4-5 pve nodes.
Storage INFORTREND connected with iscsi( multipath).
Ceph and clusterfs In the case of dedicated storage, not suitable

 

[bbgeek17]

Storage INFORTREND connected with iscsi( multipath).

So you are "wedded" to a specific hardware environment.

Ceph and clusterfs In the case of dedicated storage, not suitable

Correct, ceph is not suitable. However, if you want snapshots, then Clustered File System is your only option.
https://en.wikipedia.org/wiki/Clustered_file_system
https://manpages.ubuntu.com/manpages/bionic/man7/ocfs2.7.html

If snapshots is not a requirement, then your only other option is LVM Thick.

Note, that CFS installation, configuration and maintenance is out of scope of Proxmox as a hypervisor solution. This falls into DIY.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox