About cluster and having servers on the same network

[Diogo Jesus]

Hello everyone,
currently our company has a server at OVH (SoYouStart), which is working great for now.
Last year OVH had a major issue on 2 different datacenter which affected more than 100 000 servers. Our server was one of the affected which lead us without services for a few hours.

Also few weeks ago we had a problem in both of our disks which had to be replaced. That lead me to a full night of work. Not that I don't like but the company where I work, doesn't dispose such authorizations for that. So if we were in a cluster I could just shut it down, and proceed for the replacement.

In order to fight against those issues we are currently studying the possibility of having a cluster.

Our idea was to get a new server in Germany also from OVH (SoYouStart), which would be good since we are based in Luxembourg (between France, Belgium and Germany) and our current server is in France.


Any idea if this setup could work? I prefer to ask than buy straight the new server to find out that I just spent company money in nothing. Also OVH seems to have a large network between their datacenters but I don't know if it could be reachable for a cluster under proxmox.

Any advice?

Thank you,
Diogo Jesus

 

[mira]

If you want a cluster with one node in France and one in Germany that won't work. For cluster communication corosync is used which requires low latency (1-2ms). This means the servers have to be physically close to work in a cluster.

 

[guletz]

Any idea if this setup could work?

Hi Diogo Jesus,
You can use a possible solution:

Use a second server (S2 OVH -second data-center, S1 is what you have now) with PMX installed, but not in a cluster
- you can use ASYNC replication from S1 -> S2, let say at each 5 min(you need zfs, and pve-zsync)
- in this case, if S1 is unreachable/down, you can start the same VMs/CTs using the last succesful SYNC data
- the only problem is if you can deal with losing the last 5 min(at maximum) of data, and if S1->S2 can finish a complete SYNC in less then 5 min
- pve-zsync will replicate only the new data compared with the last succesfull sync

 

[Diogo Jesus]

If you want a cluster with one node in France and one in Germany that won't work. For cluster communication corosync is used which requires low latency (1-2ms). This means the servers have to be physically close to work in a cluster.

Thank you for the info. From what i read online apparently I can setup a vpn server to act as a "local" network between the 2 servers. Will that work?

Hi Diogo Jesus,
You can use a possible solution:

Use a second server (S2 OVH -second data-center, S1 is what you have now) with PMX installed, but not in a cluster
- you can use ASYNC replication from S1 -> S2, let say at each 5 min(you need zfs, and pve-zsync)
- in this case, if S1 is unreachable/down, you can start the same VMs/CTs using the last succesful SYNC data
- the only problem is if you can deal with losing the last 5 min(at maximum) of data, and if S1->S2 can finish a complete SYNC in less then 5 min
- pve-zsync will replicate only the new data compared with the last succesfull sync

Thank you for that option. I will study that option soon aswell.
Right now I am already running ZFS under my machine

 

[AlexLup]

There is a writeup on Proxmox and tincvpn which works, dont have the link at hand but google and you will find it.

 

[guletz]

Thank you for the info. From what i read online apparently I can setup a vpn server to act as a "local" network between the 2 servers. Will that work?

Yes you can do it with tinc as Alex Lup wrote already. But like @dlimbeck said you need a good latency. In this case you can try to use DSCP (aka trffic prioritisation) for vpn and corosync traffic(and also maybe some corosync setting for higher timeouts). But this solution will be more likely to fail. And you will get a split brain scenatio (any cluster system will need 3 hosts for avoid this case, including Proxmox)

 

[AlexLup]

There might also be another possibility, which is unicast? https://pve.proxmox.com/wiki/Multicast_notes

To note that I still havent gotten around to try tincvpn and a distant cluster yet, but am planning too, soon!

 

[Diogo Jesus]

Yes you can do it with tinc as Alex Lup wrote already. But like @dlimbeck said you need a good latency. In this case you can try to use DSCP (aka trffic prioritisation) for vpn and corosync traffic(and also maybe some corosync setting for higher timeouts). But this solution will be more likely to fail. And you will get a split brain scenatio (any cluster system will need 3 hosts for avoid this case, including Proxmox)

we can go for 3 servers without any problem. Our priority was to get the cluster to work on different datacenter. To prevent any downtime on our side. Since we mainly work with web applications, a downtime costs us a lot of money

 

[AlexLup]

This is my aim as well, but for a different purpose namely a HA NAS with all the bells and whistles like Plex for my family. To that end the plan is to install nodes on pretty crappy internet connections in my sibblings houses and to connect all that via tinc vpn, which is able to connect to several peers at the time meaning that the cluster will still be alive and well even if the main machine I host is down.

The feasibility of this is that its doable seeing the writeup I mentioned earlier where a person did just that, but I am a bit worried about corosync (proxmox cluster manager) rebooting my systems if the ping goes to high (see latest bug report for this in my history, and that is all on my internal 10gb/1gb network).

Worst come to worst I guess I will disable HA and go ceph only for storage and docker for the compute nodes such as Plex.

 

[guletz]

...as I see you are located in this nice EU, I think that using some cloud provider, and think at the GDPR, you will need to encrypt your VM/CT at least if you want to stay on the safe side(this is my own opinion). And you will also need with or without encryption to have some legal agreement about GDPR with your cloud provider

we can go for 3 servers without any problem. Our priority was to get the cluster to work on different datacenter. To prevent any downtime on our side. Since we mainly work with web applications, a downtime costs us a lot of money

For this scenario you could make a better setup. For web appl I guess you have a webserver and a database (like mysql or mariadb)

With this suppositions you can do like this:
- 3 pmx servers using 3 different data centers(1 pmx in dc1 and 1 pmx in dc2, and so on)
- you will need to have a dns for your site/s who will point to this 3 different pmx ip (round robin dns with a small ttl like 5 min)
- on each pmx you will have haproxy as frontend to the each of this http/s webserver (the local http is active and the rest are backup )
- also the haproxy will have back-end for the mysql db (local mysql active and the remote pmx as backup)
- you will need to create a mysql cluster so each instance could replicate any local write to the rest of remote mysql

Now for normal operations any mysql write (insert/delete/update ) will be replicate to the all pmx servers. For a web application most of the operation will be reading, so the replication traffic will be low

When one data center will be offline :
- the online pmx will see that one server is donwn, so will need to remove this ip from dns
- so the clients will start to use only the rest of 2 pmx hosts who are online

So on any pmx you will need to have at least 2 VMs: one for http server and 1 (better 2) with mysql

Attention, for any mysql server cluster you need a quorum like for pmx. So you will need that in the worst scenario to have 2 data center online.

Advantage for this solution:
- can scale to n data centers
- in the wors case you will lose at maximum 1/3 of your total online clients (more data center, less number clients) for a maximum of 5 minutes

Good luck!

 

[guletz]

To that end the plan is to install nodes on pretty crappy internet connections in my sibblings houses and to connect all that via tinc vpn, which is able to connect to several peers at the time meaning that the cluster will still be alive and well even if the main machine I host is down.

Alex, in our counry you can get a very good Internet ... if you are willing to pay a decent amount of money(< 10 euro for a at least 20-30 mbits).

But you will have a total different case.

 

[AlexLup]

Alex, in our counry you can get a very good Internet ... if you are willing to pay a decent amount of money(< 10 euro for a at least 20-30 mbits).

But you will have a total different case.

I live in Sweden and I have 1gbit x2 at home and they live in the sticks sitting on 100mbit each.
But I get what you are saying, I have been eyeing the 1gbit links over at OMVH as well, but since I fear for my privacy I decided its not worth it to outsource my NAS.

So I will make do with what I have!

 

[guletz]

but since I fear for my privacy I decided its not worth it to outsource my NAS

Salut Alex,

I agree 100% with you. If you want some privacy, your own home is the best you can get. For the same reason like yours I have on top of my fridge a pmx nas (I call it fridge data center )

Bafta oriunede te-ai afla!