Help with LVM over iSCSI on a 3 node Proxmox cluster

[logui]

I have a cluster with 3 nodes, each node has a second disks that I want to use to enable iSCSI then LVM on top of it, to be used as Shared storage for the cluster, similar to what Ceph will do, but I expect to be lighter on resources and logging.

I haven't been able to find a good guide on how to install and configure Open-iSCSI on each node, then enable LVM on top, and use the 3 disks as a shared storage for the cluster.

 

[bbgeek17]

Hi @logui.

It seems that you have a misunderstanding of how shared iSCSI storage works.
Unlike Ceph, which is a distributed storage meant to use local disks, iSCSI is centralized type of storage.
Meaning that a single node (or specialized cluster) is presenting single or multiple LUNs to single or multiple remote host.
Those hosts/clients can access the LUN presented over iSCSI simultaneously.

While you can arrive at a semblance of this with local disks, it's not going to be well documented anywhere because it's just not something that people do...

An example would be exporting local disks from host1 via iSCSI and then accessing the same disk from host1 via iSCSI on host1. A mouthful.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[logui]

Hi @logui.

It seems that you have a misunderstanding of how shared iSCSI storage works.
Unlike Ceph, which is a distributed storage meant to use local disks, iSCSI is centralized type of storage.
Meaning that a single node (or specialized cluster) is presenting single or multiple LUNs to single or multiple remote host.
Those hosts/clients can access the LUN presented over iSCSI simultaneously.

While you can arrive at a semblance of this with local disks, it's not going to be well documented anywhere because it's just not something that people do...

An example would be exporting local disks from host1 via iSCSI and then accessing the same disk from host1 via iSCSI on host1. A mouthful.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I understand, thank you for the clarification, any other suggestion on how to accomplish the shared storage goal with local disks without using Ceph? and not using NFS because I don't have extra hardware to mount it on.

 

[bbgeek17]

if you don't need high availability than you can use anything really.
If you need to be able to survive node failure (where storage is hosted on) then it's either Ceph or something centralized, ie outside your nodes.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[logui]

if you don't need high availability than you can use anything really.
If you need to be able to survive node failure (where storage is hosted on) then it's either Ceph or something centralized, ie outside your nodes.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I was thinking on ZFS + Replication, thoughts?

 

[Johannes S]

I was thinking on ZFS + Replication, thoughts?

This depends on your usecase. ZFS replication isn't fully synchron: You need to enable and configure the replication schedule in the vm settings, default is 15 minutes. It can be extended up to several hours (I'm not sure about the upper limit) and reduced to one minute. So in the worst case your VM will loose the data since the last sync. Now depending on the actual application this doesn't need to be a big deal (for my DNS cachers I really don't care, with a file hosting or database this might be a different story) since you might afford this minimal dataloss or design around it (e.G. by setting up a database cluster with dedicated vms on your PVE nodes without syncing them).

On the other hand you don't have a single point of failure like with a single NAS or SAS. (1)

For my homelab ZFS+replication fits my needs, Ceph would be way to much overkill. I also read in the German forum here, that for many small businesses a two-node+qdevice cluster + ZFS+replication with a schedule reduced to one minute is more than enough for their needs.

But If I ever would have to implement PVE in a professional environment I would prefer to use Ceph if possible.

Best regards, Johannes.

(1) Of course you could also put two NAS or SAS together who replicates their data between them. In such a case the single-point-of-failure argument is obviouvsly not valid anymore.