Problem with SANs on a three node cluster

[Ghosthawk]

Hello,

I’ve been trying for two weeks to set up a working solution for the following configuration using PVE 9.1.1. I’ve read dozens of articles and reinstalled the test environment dozens of times. Nowhere is a setup with two SANs described; instead, they only mention configurations with multiple internal hard drives.

We have two PVE hosts with plenty of RAM and cores, a quorum machine, and two SANs with SSDs.
The PVE hosts are identical, as are the SANs. I’m running the PVE hosts in HA mode, which isn’t a problem.
The difficulty lies in the storage connection, with the following issues.

ZFS over iSCSI seems to be extremely prone to errors. Even though both nodes and SANs are configured identically, different errors occur. The main problem is that after an HA migration of a VM, the VM can no longer access the hard drive, even though both SANs are mounted on each node. The ZFS pool name is identical on both SANs. I have tried solutions using NFS/CIFS, iSCSI/LVM, and ZFS over iSCSI. These are at least the file systems with snapshots that, according to the storage type table (https://pve.proxmox.com/pve-docs/chapter-pvesm.html), should work.

ZFS is the only system that is supposed to support replication between SANs. However, with ZFS over iSCSI, you cannot operate these mounted SANs as a RAID or cluster within Proxmox VE. When I simulate a node failure, the test VM moves, and I can neither move nor delete the VM disk (see image).

Is it better to combine the SANs into a cluster outside of Proxmox and mount only one storage?
What is the most functional setup for two nodes and two SANs?

Thank you very much, Maximus

 

[bbgeek17]

Hi @Ghosthawk , welcome to the forum.

Can you please specify what SAN type/model you are using?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Ghosthawk]

Hello,
thanks for the question!
Thats two PowerEdge R760xs with 8x 1,8TB SAS SSD for the storage, 2x 80GB for the system on a PERC H965i raid controller. The server has a two port Broadcom Adv. Dual 25Gb Ethernet (Fiber). I hope that is helpful. If you need more information, please let it me know.
Thanks, Maximus

 

[bbgeek17]

Can you clarify, did you build DIY type storage servers that are running some sort of Linux and organizes the storage with ZFS internally?
This is not an appliance type SAN?
And these two storage servers are independent of each other?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Ghosthawk]

There are two independent servers running Ubuntu 24.04, and the SSD storage on each is shared as an iSCSI target. In the Proxmox Datacenter, both SANs are integrated using ZFS over iSCSI. This seems to be working without any issues.
It is not a dedicated SAN, but rather a server with a lot of storage and iSCSI.

 

[bbgeek17]

It is not a dedicated SAN, but rather a server with a lot of storage and iSCSI.

Thank you for clarifying. There are many ZFS experts in this forum, I am not one of them. That said, I suspect that the ZFS/iSCSI plugin is sufficiently different from the Local ZFS plugin, where the ZFS replication is primarily integrated and tested. My guess is that in the current state the ZFS/iSCSI plugin is likely not compatible with the ZFS replication part.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

We have two PVE hosts with plenty of RAM and cores, a quorum machine, and two SANs with SSDs.

There are two independent servers running Ubuntu 24.04, and the SSD storage on each is shared as an iSCSI target.

a SAN would be a shared device of some sort. when you say you have "two" SANs, do you mean you have two boxes serving independent iscsi luns, or two boxes in a failover capacity (meaning one set of luns?)

In either case, this becomes a simple matter of mapping iscsi luns to your cluster.

ZFS over iSCSI seems to be extremely prone to errors.

I think, if I understand your meaning, that you would like PVE to mirror data for you across two separate iscsi LUNs. you CAN do this, but you really shouldnt at the PVE level (and, in that case, zfs wouldnt come to play at all for reasons I'll cover next.) The way to accomplish this is to map the iscsi luns (one from each filer) to the GUEST DIRECTLY, and use mdadm (or similar) to mirror them inside the guest. PVE CAN do this as an mdmirror+lvm, but due to the disconnectable design of iscsi this would be faultprone.

why should you not use nested zfs? because of the atrocious write amplification that will result , obliterating performance and overruning capacity with padding.

 

[Ghosthawk]

With a local ZFS setup, you can configure the internal hard drives into different RAID arrays—I’m aware of that.
Unfortunately, this doesn’t work with ZFS over iSCSI.

Regarding @alexskysilk’s questions:
My initial idea was to have each SAN with an iSCSI LUN, and to connect both SANs to every Proxmox node (except the quorum node). The connection is made in the data center via ZFS over iSCSI. So I have two “ZFS over iSCSI storage systems” available to each node.
The problem is that when Node 1 fails, the VMs do migrate to Node 2 (due to HA), but the VM on Node 2 fails to start. The VM disk is no longer accessible, even though Node 2 has both LUNs (from SAN1 and SAN2) attached. The issue cannot be resolved either, as the error message described in the first thread appears.

My second idea was to have each node connected to only one LUN (i.e., PVE1 to SAN1 and PVE2 to SAN2), and to replicate the VM disks at regular intervals from SAN1 to SAN2 (or, if the VM is on Node 2, from SAN2 to SAN1).
Here too, if the VM fails during migration, the VM's hard drive becomes inaccessible.

I'm not really sure how to connect the two SANs (and which file system to use) to the Proxmox cluster. ZFS with non-local storage seems to make things complicated and confusing for me. Is ZFS (over iSCSI) the right filesystem for this constellation?

What did you mean with nested zfs?
ZFS over iSCSI is not nested, one filesystem on one LUN.

Attached is an image of the ZFS over iSCSI configuration.
Thank you very much

 

[alexskysilk]

I see. so you really do have two independent iscsi hosts (targets.) There should not be any conflict between the two pools.

post logs (dmesg, iscsiadm -m session, lsblk, etc) for the host with the failure and we can go from there.

 

[bbgeek17]

Keep in mind that OP is using ZFS/iSCSI scheme. ZFS volumes on the storage sever, their iSCSI exporting, and PVE iSCSI are managed by the native storage plugin. In fact, if I remember correctly, iscsiadm is not used. This approach uses built-in QEMU iSCSI interface.

The fact that there are two "SANs" is a red herring that is distracting OP from the goal. I recommend removing one of them until all the kinks are worked out.

Start with creating a VM on each node with storage from your SAN. Then continue with testing manual migration. If all works then you can move on to node failure.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Ghosthawk]

I see. so you really do have two independent iscsi hosts (targets.) There should not be any conflict between the two pools.

post logs (dmesg, iscsiadm -m session, lsblk, etc) for the host with the failure and we can go from there.

My main problem is that I’m not sure how to integrate the two iSCSI hosts.
Option 1: Each node has one target
Option 2: Each node has both targets
Option 3: Consolidate the targets outside the cluster and connect the cluster to both nodes to ensure high availability.

I’ll try to collect the required log data today.
I’m not clear on the fundamental question of how to set this up with two iSCSI targets to achieve high availability.

Thanks

 

[Ghosthawk]

Keep in mind that OP is using ZFS/iSCSI scheme. ZFS volumes on the storage sever, their iSCSI exporting, and PVE iSCSI are managed by the native storage plugin. In fact, if I remember correctly, iscsiadm is not used. This approach uses built-in QEMU iSCSI interface.

The fact that there are two "SANs" is a red herring that is distracting OP from the goal. I recommend removing one of them until all the kinks are worked out.

Start with creating a VM on each node with storage from your SAN. Then continue with testing manual migration. If all works then you can move on to node failure.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Yeah, I should probably start by setting up the storage first and then take care of the high availability for the storage.
Just to clarify: only Ubuntu with targetcli is installed on the “SAN,” and no ZFS or other file systems are configured on the storage block device. I performed this exclusively on Proxmox Datacenter.
I’m not clear on the fundamental question of how to set this up with two iSCSI targets to achieve high availability. With one Storage (iSCSI, NFS etc.) is that pretty simple, I belive.
Thanks

 

[bbgeek17]

Just to clarify: only Ubuntu with targetcli is installed on the “SAN,” and no ZFS or other file systems are configured on the storage block device. I performed this exclusively on Proxmox Datacenter.

Ok, you can ignore everything I said. @alexskysilk is correct in the direction you should be troubleshooting.

You used a word combination that has very specific meaning in the PVE context "zfs over iscsi". However, you are just using straight iSCSI and trying to lay client side ZFS on top of it.

What you should keep in mind is that ZFS is not cluster safe. Not in the PVE environment, and certainly not when using PVE ZFS plugin which you appear to be doing based on the original screenshot.

If you insist on using client side ZFS, you are in an unsupported and unsafe territory.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Ghosthawk]

Ok, you can ignore everything I said. @alexskysilk is correct in the direction you should be troubleshooting.

You used a word combination that has very specific meaning in the PVE context "zfs over iscsi". However, you are just using straight iSCSI and trying to lay client side ZFS on top of it.

What you should keep in mind is that ZFS is not cluster safe. Not in the PVE environment, and certainly not when using PVE ZFS plugin which you appear to be doing based on the original screenshot.

If you insist on using client side ZFS, you are in an unsupported and unsafe territory.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi bbgeek17,
that has nothing to do with ignore, more with understanding.
Sorry, what did you mean with "client side ZFS"?

I thought that if “ZFS over iSCSI” (attached picture) is offered as a storage system in Proxmox, I can use it—or am I misunderstanding something? In that case, the client is Proxmox and the target is the server with the LUN. Right?
I really don't understand what I'm supposedly overlooking and how this existing feature should be used in Proxmox if the path I found at https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI is incorrect.
OK, it is one target only, maybe that is my fault.

"What you should keep in mind is that ZFS is not cluster safe." --> Thats a real good information! Thanks
So that means, ZFS is not the best solution in my case.
What is the solution von Proxmox to connect a external storage with clustering for HA. HA without redundant storage isn't a good solution.

But following your recommendation, I’m still going to set up a Proxmox HA cluster with just one external storage device.
And if that works—which it most likely will—I’ll take care of setting up the fail-safe storage required for HA.

Thank you for your support

 

[bbgeek17]

I thought that if “ZFS over iSCSI” (attached picture) is offered as a storage system in Proxmox, I can use it—or am I misunderstanding something? In that case, the client is Proxmox and the target is the server with the LUN. Right?

ZFS over iSCSI in PVE is a specific scheme that implies:
a) There is an external storage device
b) You can login to this server via SSH as root (this excludes Dell, Netapp, Hitachi, etc)
c) There is internal storage in that server (HDD, SAS, NVMe, etc)
d) The raw devices inside this storage are managed via ZFS volume manager. Whether there is an underlying RAID, or RAID implemented by ZFS is not critically important. But as you likely know, when ZFS is involved there should not be a hardware RAID.
e) There is a compatible iSCSI service as listed in the Wiki
f) PVE is able to login as root via ssh to this device, create a ZFS slice inside the device, export this slice as iSCSI LUN, map this iSCSI LUN on the PVE side, present this LUN as disk to the VM

This is a specific "ZFS over iSCSI" implementation in PVE.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

My main problem is that I’m not sure how to integrate the two iSCSI hosts.
Option 1: Each node has one target
Option 2: Each node has both targets
Option 3: Consolidate the targets outside the cluster and connect the cluster to both nodes to ensure high availability.

As I mentioned before, options 1 and 2 are available to you. option 3 is not, at least not in a non hackey way. options 1 and 2 work the same no matter whether you choose to use one, the other, or both, and dont interact/interfere with each other unless you gave them identical iqns (which must be unique.)

I’m not clear on the fundamental question of how to set this up with two iSCSI targets to achieve high availability.

I covered that here: https://forum.proxmox.com/threads/problem-with-sans-on-a-three-node-cluster.181612/post-843213
for ha, you will need to treat two luns, each on a separate target, are separate "disk drives" and use a logical mirror on top of them (mdadm.)

If you meant compute level ha, the targets don't interact and are not relevant to the function. All that matters is that each host has a path to any device hosting vm storage.

 

[Ghosthawk]

ZFS over iSCSI in PVE is a specific scheme that implies:
a) There is an external storage device
b) You can login to this server via SSH as root (this excludes Dell, Netapp, Hitachi, etc)
c) There is internal storage in that server (HDD, SAS, NVMe, etc)
d) The raw devices inside this storage are managed via ZFS volume manager. Whether there is an underlying RAID, or RAID implemented by ZFS is not critically important. But as you likely know, when ZFS is involved there should not be a hardware RAID.
e) There is a compatible iSCSI service as listed in the Wiki
f) PVE is able to login as root via ssh to this device, create a ZFS slice inside the device, export this slice as iSCSI LUN, map this iSCSI LUN on the PVE side, present this LUN as disk to the VM

This is a specific "ZFS over iSCSI" implementation in PVE.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

a) Yes, I have an external storage device
b) Yes, I can login via ssh from the PVE-Node into the storage server
c) Yes, the storage server has internal SSDs on a RAID Controller. All SSDs are in a RAID 10 without a filesystem and the LUN is configured with the hole storage.
d) "But as you likely know, when ZFS is involved there should not be a hardware RAID." --> That is new for me. I would have thought that Proxmox connects to a LUN and that it doesn't matter what it's on (with or without hardware RAID). I'll have to look into what difference that's supposed to make.
e) Unfortunately, I can't find a list of the iSCSI services. Could you please tell me where to find it?
I would have thought that iSCSI is the service itself and that the iSCSI client in Proxmox connects to the LUN provided by the iSCSI server.
Since iSCSI is defined in RFC 3720, I don’t understand what could be incompatible. But maybe there’s something I can read up on, or perhaps you have a great tip for me regarding compatibility.
f) OK, I think I understand this approach and that it basically matches what I've found in my research.
I believe the “ZFS over iSCSI” plugin connects Proxmox to the LUN on the storage device and formats the block device with the ZFS filesystem (https://en.wikipedia.org/wiki/ZFS). In Proxmox, a storage device (block device) with ZFS is then available.

I think we're misunderstanding each other a bit here. I understand the basics of iSCSI—for storage. That works fine in my environment with one storage server. The big question is, how do I use two physical storage devices in Proxmox and ensure reliability? I just don’t understand how to configure that, or if it’s even possible.
Sorry if that wasn’t clear in all my explanations.

 

[bbgeek17]

I believe the “ZFS over iSCSI” plugin connects Proxmox to the LUN on the storage device and formats the block device with the ZFS filesystem (https://en.wikipedia.org/wiki/ZFS). In Proxmox, a storage device (block device) with ZFS is then available.

You keep missing the point that the ZFS exists on the storage device side. The storage device, in your case Ubuntu, has a ZFS pool implemented by you, manually. You specify the name of that pool in PVE config to let PVE know what you named that pool.
There is no formatting with ZFS happening here, the PVE will use ZFS commands on your Ubuntu to slice a piece of ZFS Volume and then export that ZVol device via iSCSI.
ZFS as you may know has two parts - Volume Manager and Filesystem. In this case there is no filesystem, everything is Raw device.

On the PVE side it is seen as RAW LUN.

I can't find a list of the iSCSI services

https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI#:~:text=The following iSCSI target implementations are supported:

The big question is, how do I use two physical storage devices in Proxmox and ensure reliability?

You should read what @alexskysilk wrote, very carefully.


You should also accept that in Proxmox/PVE community "iSCSI over ZFS" has a very specific meaning that I have tried to explain to you.
It does not sound like this specific approach is what you want.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Ghosthawk]

As I mentioned before, options 1 and 2 are available to you. option 3 is not, at least not in a non hackey way. options 1 and 2 work the same no matter whether you choose to use one, the other, or both, and dont interact/interfere with each other unless you gave them identical iqns (which must be unique.)


I covered that here: https://forum.proxmox.com/threads/problem-with-sans-on-a-three-node-cluster.181612/post-843213
for ha, you will need to treat two luns, each on a separate target, are separate "disk drives" and use a logical mirror on top of them (mdadm.)

If you meant compute level ha, the targets don't interact and are not relevant to the function. All that matters is that each host has a path to any device hosting vm storage.

I had already configured Option 2, and of course they have different iqns. That worked fine so far (see image).
The part I'm still not sure about seems to be this:
“and use a logical mirror on top of them (mdadm.)”
I'll have to look up how to use mdadm. Thanks for the tip.

 

[Ghosthawk]

You keep missing the point that the ZFS exists on the storage device side. The storage device, in your case Ubuntu, has a ZFS pool implemented by you, manually. You specify the name of that pool in PVE config to let PVE know what you named that pool.
There is no formatting with ZFS happening here, the PVE will use ZFS commands on your Ubuntu to slice a piece of ZFS Volume and then export that ZVol device via iSCSI.
ZFS as you may know has two parts - Volume Manager and Filesystem. In this case there is no filesystem, everything is Raw device.

On the PVE side it is seen as RAW LUN.


https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI#:~:text=The following iSCSI target implementations are supported:


You should read what @alexskysilk wrote, very carefully.


You should also accept that in Proxmox/PVE community "iSCSI over ZFS" has a very specific meaning that I have tried to explain to you.
It does not sound like this specific approach is what you want.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Oh yeah, I completely overlooked the part about the ZFS pool on the Ubuntu server. You are right.
Yes, that was the list what I found and then we are on the same page, thanks. I use LIO (Linux) and it is supported.
I tested it in the test environment with one storage device, and it worked as expected. Now I’m going to work on the second storage device and take a closer look at alexskysilk’s recommendation.
Thank you very much