SAN mirroring - Storage replication

C

Claudio Martin

New Member
Jun 17, 2013
1
0
1
Hi all,

i'm testing a new proxmox VE 3.0 cluster with 2 nodes and 2 SAN ( openfiler iscsi). Each node is connected to every san with a 10Gbe with a crosslink configuration ( all is connected to all)

Node 1 Node 2
eth1 eth2 eth1 eth2
| \ / |
| \ / |
| \ / |
| \ / |
| X |
| / \ |
| / \ |
| / \ |
| / \ |
| / \ |
eth1 eth2 eth1 eth2
SAN1 SAN2

My goal is to build a full fault tolerant cluster.
I see too many cluster SAN-based with only 1 SAN ( what is the sense to build a super expensive 100 node cluster with one single point-of-failure ??), and i want to build a real-time fault-tolerant storage system.
In red-hat system you can simply configure a raid1 mirror with LVM. cmirror is a cluster component that sync DLM ( distribuite lock manager ) over network to sync raid metadata over cluster nodes.
Unfortunatelly proxmox seem not have this component (i cannot undestand why, proxmox is a red-hat cluster derived and i found all red-hat component except cmirror ). With LVM raid you can use proxmox full features with web gui, create VM, migration, etc...
Now i search some other solutions. I see RDB but if i understand well i have to build a separate cluster for RBD with is for me a crazy solution...
My object is to have a real-time replication storage in a fault tolerant cluster, what i mean is the possibility to turn off a san or unplug connection cables without any downtime ( obv timeout apart ), whitch is the one i can do with cmirror in red-hat cluster.
I'll be very happy if some proxmox developers can integrate cmirror in a future release, but for the moment anyone have a good suggestion for me ?
 
Hi, I was also bumping into this challenge but see that in the mean time (10years) this thread was not replies to.

So, our MS2019 failover cluster crashed again after an update and we're done with it. We're looking for an alternative and setup a test environment using Proxmox.

We're also having a 4-node cluster and two ISCSI SAN devices. The LUNs from both SANs are added to the pool/cluster as storage but I can only choose one to use. Oke, that may be logical, but I'll be needing a DATA replication task (mirroring) of the two SAN LUNs. And if possible an automatic failover from SAN A to SAN B (without automatically fallback).

It's for virtual machines on four 24core nodes and two SAN devices both with 10GB network interfaces. The SAN do not support ZFS over ISCSI.

Best regards,
 
Last edited:
tjarcoboerkoel said:
We're also having a 4-node cluster and two ISCSI SAN devices. The LUNs from both SANs are added to the pool/cluster as storage but I can only choose one to use. Oke, that may be logical, but I'll be needing a DATA replication task (mirroring) of the two SAN LUNs. And if possible an automatic failover from SAN A to SAN B (without automatically fallback).
Click to expand...
Proxmox does not include built-in Cluster Aware Filesystem technology beyond Thick LVM. Which in itself is not really Cluster Aware without PVE.
To use LVM you will need to present the SAN to all hosts, then LVM provision from one host. Make Proxmox aware of this setup via /etc/pve/storage.cfg. In essence the LUNs will be carved into LVM slices and Proxmox Cluster will arbitrate so that only one host is accessing any given slice at single time.

There is no requirement that LVM be placed directly on SAN device. If you feel adventurous, you can try to use some sort of mirroring technology and then place LVM on top of it. Its more than likely that this setup will require custom configuration and scripting which are beyond this forum and would put you outside PVE support realm.

Prior poster lamented that its nonsensical to place the data of multi-host virtualization cluster on a single SAN. However, a good Enterprise SAN comes with High Availability and Fault Tolerance built-in. The data protection is a given in any Enterprise SAN. Otherwise it would be a JBOD.

The best approach is to have SAN that is:
a) PVE API compatible, ie provides native Proxmox Storage Plugin
b) Contains no Single Point of Failure
c) Provides Data Protection (RAID type functionality) transparent to PVE

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi @bbgeek17 ,

Thank you for your reply.

On all servers is Proxmox PVE installed, one server is configured initial with two iSCSI LUNs from two storage devices and created a cluster on.
After that the other three servers are joined to the cluster that we created on first one.

We can see the LUNs as storage available on all four servers as "shared storage".

The SANs are from QNAP and, so perhaps I should call it NAS, as I understood it does not have support for Proxmox over ssh. It does support Microsoft hyperv and vmware. It does have cluster aware iSCSI targets. Both storage devices hold a 4disks raid5 array and has redundant network. It has no redundancy on power or its controller so that, and with the firmware updates, we have our three main risks. Perhaps replication directly between the storage devices may be a solution, but that will result in manual handling for failover (changing its IP). I dont believe the Proxmox ssh for zfs over iSCSI is supported on QNAP.


We seek:
- cluster for VM failover and load balancing
- shared iSCSI storage
- storage replication between two network storage arrays for high available storage.

What could you advice to be our best approach with Proxmox?

Best regards,
 
Last edited:
tjarcoboerkoel said:
The SANs are from QNAP and, so perhaps I should call it NAS
Click to expand...
The line between SAN and NAS is often murky. Originally, SAN meant block level protocol over dedicated non-network paths, ie FC. NAS meant NFS and CIFS. Nowadays, with network speeds reaching 400Gbits and overtaking FC - traditional SAN protocols moved over to Ethernet, as its both easier to implement and cheaper. Generally in the storage industry NAS=CIFS,NFS, SAN=iSCSI,NVMe/TCP.
tjarcoboerkoel said:
, as I understood it does not have support for Proxmox over ssh
Click to expand...
This is not one of the parameters that defines a SAN or NAS. You are thinking of ZFS/iSCSI which is a unique combination of technologies used by one of Proxmox storage plugins. As you correctly point out, it requires a) compatible iSCSI daemon b) compatible ZFS implementation c) ability to manage both via SSH. AFAIK QNAP is not one of the compatible storage devices. However there might be non-official plugins floating somewhere.
tjarcoboerkoel said:
Both storage devices hold a 4disks raid5 array and has redundant network. It has no redundancy on power or its controller so that, and with the firmware updates, we have our three main risks.
Click to expand...
Its easy and cheap to implement RAID disk protection and then export it via NFS/CIFS/iSCSI - all the pieces are available as Open Source and you can do it on any Linux server. Which is why there are so many vendors in Home and Prosumer market offering such solutions. A reliable Highly Available implementation is more expensive and usually steps up the customer in terms of hardware and software that is required.

tjarcoboerkoel said:
but that will result in manual handling for failover (changing its IP). I dont believe the Proxmox ssh for zfs over iSCSI is supported on QNAP.
Click to expand...
These two things are not directly related. Ability to use ZFS/iSCSI does not imply seamless HA on storage side.

tjarcoboerkoel said:
We seek:
- cluster for VM failover and load balancing
- shared iSCSI storage
Click to expand...
You have these two with QNap, or essentially any Linux server that exports a disk or a file via iSCSI.
tjarcoboerkoel said:
- storage replication between two network storage arrays for high available storage.
Click to expand...
You have to distinguish between site-to-site High Availability and Disaster Recovery. For HA you would traditionally have to invest into something like Netapp Snapmirror (Synchronous) or EMC SRDF-like technology.
DR, without synchronous replication, implies some data loss and change of IPs and other handles, which means manual intervention.
tjarcoboerkoel said:
I think we'll look into this direction

https://forum.proxmox.com/threads/stand-by-replication-scenario.71083/
Click to expand...
This may be your only path to achieve all desired goals within the constraints of your existing storage solution. Essentially you are building your own replication that is independent of your storage (does not depend on any unique storage functionality) and is external to PVE. As mentioned in the thread "everything is homebrew".
Make sure you have good documentation and training, to account for a "bus factor".

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
  • Like
Reactions: RolandK
You must log in or register to reply here.
Top Bottom