[ANSWERED] 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

[Frazze]

Hello guys,

I would love to get some opinions/tipps from you guys who know more than me, especially I would love to get a response from user "e100" who seems to have been running what I am going to describe below


I am currently planning on setting up a Proxmox VE Cluster trying to get HA with as low as possible and feasable money spent (I can not get a proper fencing device).

This is what my brain believes should work after some reading here (especially after some posts from e100):


Hardware:
2x "Powerfull" servers for running KVMs (with RAID10, but that is irrelevant to the following thoughts)
1x "small" server only beeing there as "quorum" witness

Goal:
nodeA RUNS KVM_1 & KVM_2 and has a syncronised working copy via DRDB of nodeB which RUNS KVM_3 & KVM_4 and itself (nodeB) has a syncronised working copy via DRBD of nodeA.
nodeC("small-only-for-quorum") is only part of the cluster to provide Quorum so I can have VMs manged via HA failover-domains.

Setup:
On the 2 "powerfull" servers (nodeA & nodeB) I want to install Proxmox on top of Debian so I can do manuell partitioning.
I would create on each of the two "powerfull" nodes:
3x LVM2 VolumeGroups(VG):
1) VG_host (just containing debian proxmox host system)
2) VG_vms_local (containing the KVMs which should be RUNNING on the host when the cluster is fully operational, meaning the opposite node is NOT down)
3) VG_vms_remote (containing the KVMs which should ONLY be running when the opposite node is down, meaning at that time there would be runnung all 4 KVMs on the node)
VG_vms_local and VG_vms_remote would be replicated via drbd, where:
VG_vms_local is drbd ressource r0
VG_vms_remote is drbd ressource r1
drbd replication is done via a set of two separate network links (eth2 & eth3) by bonding, directly, back-to-back
cluster / LAN communciation is done via a set of two seperate network links (eth0 & eth1) by bonding, to ONE switch

On the "small" server (nodeC) I would do a standard Proxmox install.

Now all 3 servers would be added to one Proxmox cluster.


How I think it would run:
With all 3 servers beeing in the cluster and only nodeA and nodeB hosting KVMs (replicated via drbd to each others) and the third node3 only beeing there as "Quorum witness" (having no access to the KVMs data), I should be able to run HA.

My questions now:
I need to setup proper failover-domains in case nodeA fails (no connection to the cluster for whatever reason).
Now nodeB & nodeC should both see that nodeA is not responding and therefore the two KVMs KVM_1 & KVM_2 should be started AUTOMATICALLY on nodeB.

1) Can this be done via failover-domains? How to do it? Is there a good example/howto into this failover-domain thing? This is where my knowledge/understanding gets blurry.

2) Let's assume everything worked out as described (example above), what happens after nodeA comes back up again? Will KVM_1 & KVM_2 be stopped on nodeB and started on nodeA again AFTER drbd syncronisation has been completed, automatically?

3) Can I really do this setup without a fencing device that ENSURES that (in the above example) nodeA IS down?
What if nodeA only lost network connection to the cluster / LAN on eth0 & eth1 and the drbd replication connection via eth2 & eth3 is still active? It would mean I have a fucked up mess, right? Can this be taken care of by failover-domains?

4) Did I forget anything? Is my setup build on any faulty assumption that would break everything I have planned?




Thanks for reading the whole post up to this last line

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

*Bump*
None? Or just too much text...?

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Sorry for bumping again, but it's hard to believe my questions really are that hard?
Come on, there has to be at least a few people having done or tried at least thought about the same thing?!

 

[tom]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

search the forum, your setup (similar) is discussed multiple times already. in short, DRBD is not integrated in the Proxmox VE HA stack.

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

search the forum, your setup (similar) is discussed multiple times already. in short, DRBD is not integrated in the Proxmox VE HA stack.

Thanks Tom, but I know it has been discussed multiple times, I read here before asking questions in general, the whole idea of the setup is coming from those discussions, but the questions I asked I am still not 100% sure, especially this one:

3) Can I really do this setup without a fencing device that ENSURES that (in the above example) nodeA IS down?
What if nodeA only lost network connection to the cluster / LAN on eth0 & eth1 and the drbd replication connection via eth2 & eth3 is still active? It would mean I have a fucked up mess, right? Can this be taken care of by failover-domains?

 

[e100]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

If you want to use HA for having the VMs failover automatically do not use DRBD.
There is one failure senerio where this can result in bad things happening.

I feel more comfortable allowing a human to make the decision on where to start a VM when something goes wrong.

Also, you really do want a proper fencing device for HA.
I see old APC masterswitches on ebay sometimes for as little as $100 US

Imagine what would happen if the same VM was running on both nodes at the same time.
Your filesystem for that VM would be corrupted beyond repair, without fencing you turn HA into MD (Major Disaster)

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Thanks for your answer e100.

So it is as my question already implied, it cannot be done in a secure way without a real hardware fencing device.
What a pity :/

So I will only do drbd 2 nodes in primary/primary mode and do start the VMs manually on the nodes in case of defect.

To be sure, before starting the VMs on the still working node, I will disable the drbd network interface on the still working node so I am sure the defect node can not write crap and give me a major disaster.

Any further suggestions to this setup beyond my plan?

Thanks again!

 

[udo]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Thanks for your answer e100.

So it is as my question already implied, it cannot be done in a secure way without a real hardware fencing device.
What a pity :/

So I will only do drbd 2 nodes in primary/primary mode and do start the VMs manually on the nodes in case of defect.

To be sure, before starting the VMs on the still working node, I will disable the drbd network interface on the still working node so I am sure the defect node can not write crap and give me a major disaster.

Hi,
this is normaly not nessesary - if one node is defective, your drbd-ressources are standalone. Only if one node has an huge problem (between life and death) you can disconnect the drbd-resourcen.

Any further suggestions to this setup beyond my plan?

Thanks again!

if you use the third node (from your first plan), you don't loose the quorum in such an case (one node fail). Otherwise you must first adjust the quorum before you can move an VM-config and start this VM on the remaining node.

Udo

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Thanks for your answer Udo!

if you use the third node (from your first plan), you don't loose the quorum in such an case (one node fail). Otherwise you must first adjust the quorum before you can move an VM-config and start this VM on the remaining node.

What would be the advantage of that 3rd real "dummy" node in this case against the

<cman two_node="1" expected_votes="1"> </cman>

or the

#pvecm expected 1

solution?

Can there be any problems if the expexted_votes count is always set to "1" in normal cluster operations?
What would be the benefit of an 3rd real node in this setup vs the simple "hack" of expected_votes=1?


Thanks again guys!

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

*push* ( sorry )

 

[udo]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Thanks for your answer Udo!



What would be the advantage of that 3rd real "dummy" node in this case against the

<cman two_node="1" expected_votes="1"> </cman>

or the

#pvecm expected 1

solution?

In case of emergency you don't need change something first to reach the quorum.

Can there be any problems if the expexted_votes count is always set to "1" in normal cluster operations?
What would be the benefit of an 3rd real node in this setup vs the simple "hack" of expected_votes=1?
Thanks again guys!

With expected votes 1 you can have both nodes running with "valid" quorum also in case of an broken network connection... this make no sense for me.

Udo

 

[Frazze]

Re: 2-Node-HA-drbd plus 1 small-Node for Quorum - Cluster Setup

Okay, thanks, I'll mark this as solved than, thanks again guys