HA cluster with 2 nodes setup

[menelaostrik]

Hi,
i have decided to use 2 hp G9 servers and setup a 2 nodes HA cluster for production(at the moment we cannot afford a 3rd node)
and will use a VM hosted on a seperate server, on another DC(around 6-7ms ping) that will keep the quorum with corosync-qnetd

The servers feature 2 NICs with 4 ports each(8 ports total) and a management port running ilo advanced

Here is what we did in a few words:
1) Bonded 7 of the ports to use for replication(using zfs) which are connected directly from server-to-server using CAT6e cables using balance-rr (which for a wierd reason is capped at 2.25gbps).
2) The last port is connected to the internet switch and used for internet on both machines.
3) The ilo port is connected to a seperate network(which is behind dnat firewall in order to harden security).
4) The VM used for quorum, accesses the servers on the Internet port.

The scenario seems to be working currently in the lab but i wanted to make sure that my thinking won't get me in trouble after i ship the servers to the datacenter.

1) What do you think of the setup?
2) do i need to configure fencing for the ilo ports?
3) is there anything i could do better?

thank you in advance for your time,
Menelaos

 

[ph0x]

1) no one will encourage this setup, so good luck as long as it's working. Especially no one can tell if the lab results are transferrable to a DC environment.
2) no, Proxmox HA works with an own watchdog based fencing, you don't need to configure anything.
3) get a third node!?

As far as I know from what I read, HA won't work with local storage and Ceph is out because of two nodes. So how do you manage the storage?
Balance-rr often leads to resending packets because of them arriving in the wrong order. So 2.25 GBit is probably as good as it will get.
All in all this sounds like a home lab approach rather than a production environment.

 

[menelaostrik]

1) no one will encourage this setup, so good luck as long as it's working. Especially no one can tell if the lab results are transferrable to a DC environment.
2) no, Proxmox HA works with an own watchdog based fencing, you don't need to configure anything.
3) get a third node!?

As far as I know from what I read, HA won't work with local storage and Ceph is out because of two nodes. So how do you manage the storage?
Balance-rr often leads to resending packets because of them arriving in the wrong order. So 2.25 GBit is probably as good as it will get.
All in all this sounds like a home lab approach rather than a production environment.

Could you please elaborate why the home lab results cannot be transferred to a DC?

the storage is managed by the inbuilt proxmox replication over zfs

 

[ph0x]

Let me see ...
- do you have any SLA that guarantees you that the quorum device will always be reachable?
- do you have any SLA that guarantees you a maximum ping between the two datacenters (6-7 ms is not really LAN speed anymore, knowing that the qdevice doesn't necessarily need that)?
- do you have a plan if the second machine goes down while the other datacenter is unreachable?

It is and will not be production safe, but you can do whatever you want.

And HA does work with storage replication? Did you test that? Every source says to have shared storage for HA, but it doesn't mean that it maybe can't work otherwise.

 

[menelaostrik]

Hi
Their sla for the connectivity between the dcs (they are both from the same company and have direct links between them) of 99.99%
and max ltency at 9ms
My tests has shown latency of no more than 7ms though.

if connectivity it lost between the dcs and one node goes down, the ha will not work, at least not until the connectivity is back. But since it’s very rare for both to happen at the same time, I can live with that.

replication is occurring every minute so if a node goes down the maximum amount of data I will have lost will be 1 minute which is something I can also bear.

 

[sidereus]

Put a Raspberry Pi as a quorum device in a place where 2 nodes are located.

 

[menelaostrik]

Put a Raspberry Pi as a quorum device in a place where 2 nodes are located.

Do you believe that our scenario will be trustworthy if we use a raspberry Pi to keep the quorum?

i have no experience with raspberry but i just found out that i can easily install debian on it so i could also use it as a firewall for the ilo management ports.

 

[sidereus]

Do you believe that our scenario will be trustworthy if we use a raspberry Pi to keep the quorum?

i have no experience with raspberry but i just found out that i can easily install debian on it so i could also use it as a firewall for the ilo management ports.

Why not. To use it as a firewall you will need also an USB ethernet adapter, because there is only 1 ethernet port.