2 Proxmox Hosts and 2 iSCSI TrueNAS servers setup

[Moditproxer]

I am trying to have an HA setup where I have 2 Proxmox Hosts and then 2 iSCSI TrueNAS servers as the data (VM) storage. I have the iSCSIs set up to connect via The Datacenter->Storage but I want the storage to be redundant to each other. I was hoping I could create a Raid1 or something similar so that if one storage went down the other would still be usable and the VMs would be able to stay up.

Does anyone know if this is possible? I see the iSCSI over ZFS but that doesn't seem to have the option to do an iSCSI Raid 1 from 2 different iSCSI devices. Anyone's help would be greatly appreciated. Yes I do have a witness server for the HA but it is just a Linux system running the required 2 quorum packages.

 

[bbgeek17]

Without going into details on whether the desired solution is technically sound, there is no built-in mechanism in PVE to create RAID on top of two iSCSI disks in a shared storage environment. You will need to do so manually, perhaps via mdadm.
How the entire end-solution will function under load is questionable. I think its safe to say that very few people attempted something like this, and likely no one runs such a setup as production.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Moditproxer]

Is there a way to do this that is not in a RAID 1 type of setup but instead has some sort of syncing going on in the storage?

 

[bbgeek17]

If you clarify your requirements, then it will be easier to give more direct answers.

- You said you have two nodes in your cluster. Assuming just one storage device for now - do you want to use it as shared storage?
- If you do want shared storage, have you researched your option for shared storage? (LVM, OCFS2, ISCSI over ZFS, other?)
The only built-in replication in PVE, at this time, is ZFS based replication. ZFS is not suitable for shared storage.
Anything else, you will have to architect yourself. One option would be to use built-in TrueNAS replication, which would be completely unaware of your application (PVE).

In summary: given all the input facts so far, there is no good plug-and-play solution for what you want.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Moditproxer]

Right now I added both of the storage devices to the 2 cluster environment as iSCSI devices. If I use ZFS replication between the two of them what would it take to be able to start a VM up if the primary storage goes down so we can use the replicated VMs on the secondary storage which is already connected via iSCSI to the PVE environment? Would I have to rebuild them as far as letting PVE know that they are on the other device?

I can't believe that people think HA is only for the controllers and not for the storage as well. 2 host nodes in the cluster with one storage still has one huge point of failure. I've experienced it already with VMware when one of the storage nodes went down and wasn't in a RAID1 configuration via HPs Storage Manager software. Luckily the other part of that physical hardware that went down was in a RAID1 with another storage server and we didn't lose any of those VMs or data.

 

[bbgeek17]

Right now I added both of the storage devices to the 2 cluster environment as iSCSI devices. If I use ZFS replication between the two of them what would it take to be able to start a VM up if the primary storage goes down so we can use the replicated VMs on the secondary storage which is already connected via iSCSI to the PVE environment?

The best way would be to try it. And to point out again - if you use this method you will not be able to use your storage as Shared storage, ie with instant live migrations.

I can't believe that people think HA is only for the controllers and not for the storage as well.

I dont think anyone has stated so. Most commercially available storage solution provide HA as a given, including Blockbridge. You can run TrueNAS as HA solution as well. You may need to invest in appropriate hardware to do so.

There are, of course, Open Source solutions that provide HA at storage level as well. Ceph being one of them, but its not suitable to your infrastructure.

I think your challenge is that you are trying to find a cookie-cutter solution constrained by a non-production mix of hardware (i.e. two node PVE cluster is not recommended due to split-brain possibility, two independent TrueNAS servers that are not PVE aware).

Ideally, for a reliable production system one defines requirements and then invests in hardware.

If this is a home-lab, then you have to use a mixture of solutions at your own risk.

Best of luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox