shared storage over WAN

[AlBayrouni]

i have a proxmox cluster with 3 nodes (the number could scale in the future) and these nodes are connected over WAN through public IP and they are located in different geographical location and i want to make a shared storage for this cluster to get east and fast VM migration between nodes and ensure high availability

i wanted to know if it's possible in my case to do the shared storage over WAN ?
if yes which type of storage do you recommend (zfs, nfs .....)?
thanks you so much

 

[Dunuin]

I guess best performance you should get with ZFS as thats not a real shared storage. Its just local storage that gets synced once per minute or so giving you something very similar to a real shared storage like NFS or ceph, but without the big latency across countrys/continents, that is still able to do HA. But downside would be that you need alot of disks as everything is just mirrored across nodes. Lets say you want 2TB of storage with redundancy and you got 5 nodes. Then you would need to buy 10x 2TB SSDs to be able to create 5x mirrors/raid1 resulting in just 1.6TB of total usable storage. And another downside would be that the ZFS pools would never be perfectly synced. This is no problem when migrating, while all nodes are running, which is still very fast as only up to the minute needs to be transfered. But in case a node fails that local ZFS storage isn't accessible anymore so HA can't sync the latest data and will start the failed VMs on another node but with data that might be a minute old. So in case of a failing node you basically loose 1 minute of data.

You wouldn't have that problem with ceph but as far as I know performance won't be great doing sync writes across continents with latencies of milliseconds instead of microseconds.

 

[LnxBil]

Hopefully, you're "only" a few km away, then you could go with HA storage systems (with internal replication) but it works only for a very small latency. If you exceed that latency, corosync will act up. It is not recommended to build a WAN cluster but it'll work until a certain point.

 

[Neobin]

About what WAN line speeds (both directions) per node do we even talk?

 

[LnxBil]

About what WAN line speeds (both directions) per node do we even talk?

What do you mean by speed? It should obviously be light, so roughlty 150-200 Mm/s in optic fiber, so for longer distances, the latency is only defined by that speed. In corosync, the cluster breaks apart if you have over 50ms delay, so you can compute how far away those nodes can be, this is only for the cluster communication, which is all Ethernet. If you have a dedicated storage network, you will have the same minimum delays and added the storage delay. You can simple compute the maximum IOPS you can archieve and also the throughput (writing, reading should ALWAYS be local).

 

[Neobin]

What do you mean by speed?

I mean the WAN speed/bandwidth with which the OP's 3 nodes are connected to each other.
For example: Do we talk about 3 nodes in 3 different datacenters with a (guaranteed?) 1 Gbit/s connection each or do we talk about 3 nodes in 3 different private homes with, for example, only 50 Mbit/s upload?
Should be an important information for considerations about some sort of shared/replicated storage, no?
Depending on the OP's requirements and amount of changing data (which we also not really know), maybe not even a replication could fit his needs because of a too low WAN (upload) bandwidth; replications get too old, for example.

 

[LnxBil]

Should be an important information for considerations about some sort of shared/replicated storage, no?

Yes, sure this is important, but having a working cluster is more important. If the replication delays the cluster/corosync packages, you will have fencing, which is also not good. You need QoS of some sort to ensure that the fencing does not happen.

 

[AlBayrouni]

I guess best performance you should get with ZFS as thats not a real shared storage. Its just local storage that gets synced once per minute or so giving you something very similar to a real shared storage like NFS or ceph, but without the big latency across countrys/continents, that is still able to do HA. But downside would be that you need alot of disks as everything is just mirrored across nodes. Lets say you want 2TB of storage with redundancy and you got 5 nodes. Then you would need to buy 10x 2TB SSDs to be able to create 5x mirrors/raid1 resulting in just 1.6TB of total usable storage. And another downside would be that the ZFS pools would never be perfectly synced. This is no problem when migrating, while all nodes are running, which is still very fast as only up to the minute needs to be transfered. But in case a node fails that local ZFS storage isn't accessible anymore so HA can't sync the latest data and will start the failed VMs on another node but with data that might be a minute old. So in case of a failing node you basically loose 1 minute of data.

You wouldn't have that problem with ceph but as far as I know performance won't be great doing sync writes across continents with latencies of milliseconds instead of microseconds.

Thanks you so much for this interesting strategy,
i have another suggestion and i appreciate your thoughts toward it
- in the officials docs, in the table bellow,
it's mentioned that zfs could be shared over iscsi
is it a good option ?

 

[AlBayrouni]

Hopefully, you're "only" a few km away, then you could go with HA storage systems (with internal replication) but it works only for a very small latency. If you exceed that latency, corosync will act up. It is not recommended to build a WAN cluster but it'll work until a certain point.

thank you so much for mentioning this, it's a important point to consider

 

[AlBayrouni]

@LnxBil @Dunuin
i want to thank you so much for your interesting interventions, i am really still struggling to find the most convenient storage option i could do for my "over WAN" cluster
i have another suggestion(that will compromise high availability) and i appreciate getting your thoughts about it

What about :
- having a glusterFS shared storage (over WAN also) between proxmox nodes to make migration of VMs between nodes faster
- keep using local zfs storage for making snapshots (as glusterFS don't have the functionality of making backups)
- adding a proxmox backup server node for storing backups of my VMs across my nodes
is it the best strategy for an infrastructure over WAN ?

 

[LnxBil]

is it a good option ?

over WAN? No ...

If you have site A and the SAN is on site B, you will loose everything is internet of A is down or internet of B is down or the SAN is down. This has nothing to do with any kind of HA setup. You could however use #sites SANs and synchronise over them, but this will be the same as using one node per site and do ZFS replication.

 

[LnxBil]

- having a glusterFS shared storage (over WAN also) between proxmox nodes to make migration of VMs between nodes faster

This is a dedicated shared storage, whereas all data is written at all nodes, which is a good cluster setup, but it may be very slow depending on the throughput and AFAIK this is a synchronous setup, so it is time critical but will yield the most up-to-date data.

- keep using local zfs storage for making snapshots (as glusterFS don't have the functionality of making backups)

a backup is not a snapshot and you will still be able to use snapshots for QCOW2 on glusterfs. It's just another layer. ZFS has the advantage of asynchronous replication, which will yield much better write performance whereas you will not have up-to-date data on all other nodes.

- adding a proxmox backup server node for storing backups of my VMs across my nodes
is it the best strategy for an infrastructure over WAN ?

It's always good (or say required!) to have off-site backups. This is also asynchronous, so you're limited by your throughput.