4+ node management cluster with DRBD pairs: Where to do fencing for HA?

[Pegasus]

Hello.

I have a ProxMox management cluster that currently has 5 nodes (it will have 6 eventually) and I plan to set up DRBD to mirror data between machine pairs whith a goal of high-availability (via automatic live migration.) But where should I implement fencing? In DRBD, in ProxMox, or both? (I'll be using HP iLO as the fencing devices.) The wiki isn't clear, because while I have plenty of nodes to satisfy quorum, the storage is not shared between them all, which is the only situation the wiki seems to address.

Thanks for your time! And many many thanks to the ProxMox team for their great and hard work!

 

[joshin]

I have some similar questions on Proxmox fencing. (I'm experienced with fencing, but not with doing it on Proxmox.)

My suggestion is to implement fencing on Proxmox, but only on the nodes attached to the shared storage. If a node isn't attached to shared storage, it doesn't do you any good to be in the HA group.

Obviously, you need some way to ensure that your DRBD pair does proper failover. That may require it's own implementation of fencing to ensure availability to the nodes.

 

[e100]

You need to configure failover domains for the DRBD nodes, then place HA VMs into the failover domains.
This will ensure that the HA VM will only run on the proper DRBD nodes. You can also set preferences in failoverdomains, if both nodes are up the VM will run on the preferred node.

A forum search should be suffecient to find all the necessary information.

Only one caution.
Proxmox HA is not aware of DRBD status.
If HA kicks in while your DRBD is already split-brain you may have a very bad day.
For this reason I am not using HA with DRBD and have choosen to always let a human make the appropriate decision.

We need some method to prevent HA failover when DRBD is split-brain, I have not researched how to do this since it is not important to me at this time.
Anyone using DRBD with HA should really figure this out before going into production, IMHO.

 

[Pegasus]

My suggestion is to implement fencing on Proxmox, but only on the nodes attached to the shared storage.

But in my case, I'll have three separate shared storage devices with two nodes attached to each. Each will need HA, but only within its pair. From what I read, I should be able to do this with failover domains, right?

Proxmox HA is not aware of DRBD status.

Feature request, please, ProxMox developers!

If HA kicks in while your DRBD is already split-brain you may have a very bad day.

Would setting DRBD to just disconnect the resource(s) on split-brain detection (therefore requiring human intervention) be sufficient?

I'm also not completely clear on what exactly it means to set a storage item as "shared" in the ProxMox GUI. My current understanding is that either node must be able to read & write to it at the same mount point and that concurrency is managed by the underlying subsystem, DRBD in this case. Is this correct?

How do you have your ProxMox/DRBD/manual "HA" cluster configured, e100?

 

[e100]

I have fencing setup for all my nodes, good if a machine locks up not just for HA.

Each DRBD node has two DRBD volumes VMs that usually run on node A use DRBD0 where VMs for node B use DRBD1. This helps when you have to recover from split-brain, all explained in the wiki.

failoverdomains for my DRBD volumes look like this: (I have them setup just not using them right now, yes I have tested this before)

........
  <rm>
    <failoverdomain name="vm1-vm2" restricted="1" ordered="1" nofailback="1">
      <failoverdomainnode name="vm1" priority="1"/>
      <failoverdomainnode name="vm2" priority="100"/>
    </failoverdomain>
    <failoverdomain name="vm2-vm1" restricted="1" ordered="1" nofailback="1">
      <failoverdomainnode name="vm2" priority="1"/>
      <failoverdomainnode name="vm1" priority="100"/>
    </failoverdomain>
  </rm>
</cluster>

A VM assigned to failoverdomain VM1-VM2 would prefer to run on VM1
A VM assigned to failoverdomain VM2-VM1 would prefer to run on VM2
The HA VM in the failover domain can only run on the nodes specified within the failoverdomain.

I put LVM on top of DRBD, then mark the LVM volume as shared.
In the GUI you can specify the specific nodes that have access to the shared volume, so I only select the two DRBD nodes that have access to that volume.

Setting DRBD to disconnect on split-brain I do not think would solve anything and would likely create more problems.
Would you want to disconnect the storage from a running VM? Most likely not.

Split-brain is usualy caused by human error or network issues (humans accidently unplugging stuff).
It will happen when you least expect it and if you are not prepared for it will bit you in the rear hard.

AFAIK the only issue with DRBD and HA is if a split brain happens, then some time later HA kicks in before you correct the split-brain.
You have three choices in this situation:
a) discard all data stored between split-brain and HA failover event.
b) discard all data stored after the HA failover event
c) manually try to merge the two divergent paths

None of those are acceptable, IMHO, for anything with the name HA in it.

Provided you know this and have good monitoring in place to alert you to DRBD split-brain then maybe you are ok using HA with DRBD.
Myself, I prefer to make the decision on when to failover because A, B and C are not decisions I ever want to make or have to explain to my boss/customers.

 

[cesarpk]

Hi e100

Your post don't show the "pvevm" and "failoverdomains" directives in the cluster.conf file.

I have a setup very similar, but with some problems, can you see my post in this link and answer me? ... i will be very grateful
http://forum.proxmox.com/threads/18609-Problems-with-a-VM-that-is-in-HA

Best regards
Cesar