HA in only 2 nodes without documentation on Proxmox website

[cesarpk]

Hi All

Pelase, can anyone help me?

I need configure proxmox with:
- Only two nodes (I know configure proxmox in cluster (not to be confused with Cluster HA))
- DRBD ( I know DRBD very well for configure primary/primary and primary/secundary)
- Extra disk for DRBD in each node
- KVM (not openvz)
- Without fence devices
- Without storage external (NFS, iSCSI, etc.)

The idea is that if i have 2 nodes with VMs are powered on, and in case of breakdown of a node, manually start the VMs on the other node

And I need to know:
- How configure Proxmox for do it?, ideally with and without fence devices (excluding the part "fence devices", that I know how do it, only the part of configure of proxmox)

References:
- Because there is no tutorials for this case
- And neither for use with only 2 nodes:
On the website:
http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
literally says: "For this testing configuration, two DRBD resources were created, one for VM images an another one for VMs users data", but does not say how to do it
And on the website:
http://pve.proxmox.com/wiki/DRBD
no says how configure VMs users data
Then, there is no tutorials for 2 nodes

- For my test, when I turn off a Proxmox Cluster node, the other Proxmox Cluster node is automatically disabled, and can not be used it the second node until it is restarted, so it is impossible to test for faults.

I hope someone can help me

Best regards
Cesar

 

[e100]

If you search the forums you will find that the bits and pieces of this have been discussed before.

The first wiki link you provided mentions:
To be able to create and manage a two-node cluster, edit the cman configuration part to include this:

<cman two_node="1" expected_votes="1"> </cman>

That fixes the node being disabled issue you mentioned.
But I highly recommend you do not run a production HA cluster like that.

You can not have proper quorum with just two nodes.
For HA to actually be HA you need proper quorum and fencing so you need at least three nodes.
The third node could be some tiny low powered Intel ATOM CPU server, it does not need to run VMs, just install proxmox and join it to the cluster.
It will work fine to provide proper quorum.

Read this whole thread but these two posts specifically:
http://forum.proxmox.com/threads/8693-is-there-an-alternative-to-fencing-for-HA?p=49323#post49323

http://forum.proxmox.com/threads/8693-is-there-an-alternative-to-fencing-for-HA?p=49835#post49835

After reading that you will now know why you must have fencing.
Then stop to think for a moment, how can you properly fence with only two servers?
Now you know why you need three nodes even if only two of them are doing any real work.

Can you make it work without fencing and only two nodes?
Yep, you can do the expected_votes hack and make fake fencing devices.
But you end up with a DOA (Disastrous Outage Awaits) instead of HA(High Availability)

If you want HA do it right or do not do HA at all.
You can do a two node DRBD cluster, without HA, for example.

 

[udo]

Hi All

...
References:
- Because there is no tutorials for this case
- And neither for use with only 2 nodes:
On the website:
http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
literally says: "For this testing configuration, two DRBD resources were created, one for VM images an another one for VMs users data", but does not say how to do it
And on the website:
http://pve.proxmox.com/wiki/DRBD
no says how configure VMs users data
Then, there is no tutorials for 2 nodes
...

Hi Cesar,
most things was explaind by e100. About the two resourses:
I would also strongly recomment two resources, but both resources for vm-images (normal lvm-storage on top of drbd). For each node one resource.
E.g. node a use the lvm-storage a_sata_r0 (on drbd r0) with VM-IDs 100-199 and node b use the lvm-storage b_sata_r1 (on drbd r1) with VM-IDs 200-299.
In this case you can easily see, if all VMs running on node a use the "right" storage and vice versa. If you run VMs on one node which used both drbd-resources (and the same resource is using on the other node) you run in trouble if a splitbrain condition occured. Because you can't resync the drbd to one side without overwriting the data of the VMs which are running before there.

To configure two resources see here http://pve.proxmox.com/wiki/DRBD and use a second disk (or raid partition) for r1 like r0 before (sdc1 instead of sdb1 and port 7789 instead of 7788).

But you wrote, you have experiences with drbd - so perhaps you allready know that (and have this also configured).

Udo

 

[cesarpk]

Udo and e100 thanks for your patience to explain me, learning is very nice thank to you
I apologize if I do a lot of questions but I try to understand how to set "Proxmox" correctly

Please let me ask you some questions:
(My question is strategic to know how build the DRBD cluster and issues related to Proxmox)

I understand that I need at least 3 nodes for use "fence device".. ok! (thank you e100)

But if i use only two nodes with drbd and without fence device, and i want manually to do in this order:
1- in case of breakdown of a node, manually disconnect it from the electric power
2- Start the VMs in the other node that is active with a minimum steps

1- is it possible?
2- DRBD must be in active/active for all resources? (thanks Udo for explain me "one resource (r0,r1) for each node" i understand you very well and how to avoid problems for breakdown.)
3- What happend with configure VMs users data?
(this says in http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster )
4- Is possible configure Proxmox for that with a minimum of steps manually start the VMs in the other node?
5- How i configure Proxmox for that with a minimum of steps manually start the VMs in the other node?

Waiting for your early reply and thanking you for all your attention, I say goodbye

Best regards
cesar

 

[cesarpk]

Hi People

Thank you e100 and udo for enlighten me
Please e100, udo or anybody can help me ??????

apology if I am in the right way, but Mr. Google does not help me)

I have configurated proxmox ve 2.x with:
- Only two nodes (I know configure proxmox in cluster (not to be confused with Cluster HA))
- DRBD (installed and functional, and I know DRBD very well for configure primary/primary and primary/secundary)
- Extra disk for DRBD in each node
- KVM (not openvz)
- Without fence devices
- Without storage external (NFS, iSCSI, etc.)

The idea is that if i have 2 nodes with VMs powered on in both nodes, and in case of breakdown of a node, manually start the VMs on the other node applying a technique of manual fence (fencing proven that it works)

And I need to know:
For drbd:
1- DRBD must be primary/primary or primary/secondary? (i will use one storage resource DRBD (r0, r1) per node)
2- Should i use LVM on top of resource DRBD or what?
3- Did any other details I should know?

For Proxmox VE 2.x:
4- Should i configure LVM group on Proxmox GUI or what?
5- Did any other details I should know?

Best regard
Cesar

 

[e100]

Primary/Primary
LVM on top of DRBD, then add the LVM volumes in the GUI so you can use them.

Other things you should know?
I pretty much set things up like it states here: http://pve.proxmox.com/wiki/DRBD and use two DRBD volumes like you plan to do.

When you add the DRBD LVM volumes in the GUI, mark them as shared and also select the specific nodes they belong to.
That way if you ever grow your cluster these DRBD volumes will only be available on the two nodes.

Naming is important to prevent confusion.
We name our nodes vm1,vm2,vm3,vm4 etc.
So we name our LVM volumes like this:
vm1-vm2 <-stores disks for VMs that run on vm1
vm2-vm1 <-stores disks for VMs that run on vm2

That way looking at the storage name I know where the VM primarily runs and what two nodes it can run on.

Since you have only two nodes be sure to change the expected_votes in cluster.conf as mentioned before.

When something goes wrong before starting a VM on the other node be 100% sure that the VM is not still running on the other node.
That is what fencing with 3 or more nodes does with the HA setup, but without fencing you need to make the checks yourself.

I appreciate the thanks from you and others.
Proxmox is a great platform with a very helpful community.
Sharing my knowledge and helping others is my way of giving back and contributing to this great open source software.

 

[cesarpk]

Primary/Primary
LVM on top of DRBD, then add the LVM volumes in the GUI so you can use them.

Thank you e100 for your time and sharing your knowledge
for me is very important !!!, hopefully later I can also help those in need, as now you do it me.
And by my experience, I call to the recognition of e100 and udo for the great effort devoted generously to help those in need!!!
This world would be different if everyone were like them

Now, going to my problem:
I changed the expected_votes in cluster.conf as you mentioned before, and work very well

if you'll excuse me, two parts are not clear:

1- When you say: "add the LVM volumes in the GUI", i beleive that as can be seen in the Proxmox GUI you mean to "Panel Datacenter - Tag storage - button add - option LVM Group", correct?

2- When drbd is completely configured, for each storage resource DRBD on the porxmox GUI should I enter the two node names and not only one node name, correct?

A question that intrigues me for a long time:
3- On the website:
http://pve.proxmox.com/wiki/Two-Node...bility_Cluster
literally says: "For this testing configuration, two DRBD resources were created, one for VM images an another one for VMs users data"

So, What is VMs users data?, what utility has? and how configure it?, because it not says more about it

Best regards
Cesar

 

[cesarpk]

Ah e100, i forget it:

For drbd on the scenery explained in the previus post, What is the best configuration for after-sb-2pri.?
call-pri-lost-after-sb, disconnect, violently-as0p or what?

I think that not stop the work by DRBD disconnection is the best, or not? ¿what do you think?

Best regards
Cesar

 

[e100]

I have all of my DRBD volumes configured with:

    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;

Disconnecting is the best solution, in my opinion, because your VM can continue running without disruption and a human can later decide how to clean up the mess.
We use monitoring software to alert us when there are problems so we get a message within a few minutes of a DRBD split brain happening.

It is also fairly simple to recover from a split brain, when using two DRBD volumes, as I outlined here:
http://pve.proxmox.com/wiki/DRBD#Recovery_from_split_brain_when_using_two_DRBD_volumes

 

[cesarpk]

Thank you very much e100, I will follow your advices

And finally if not is much hassle, could you answer my previous post?
http://forum.proxmox.com/threads/94...entation-on-Proxmox-website?p=53311#post53311

Best Regards and God bless you.
Cesar

 

[e100]

1- When you say: "add the LVM volumes in the GUI", i beleive that as can be seen in the Proxmox GUI you mean to "Panel Datacenter - Tag storage - button add - option LVM Group", correct?

Yes that sound about right.

2- When drbd is completely configured, for each storage resource DRBD on the porxmox GUI should I enter the two node names and not only one node name, correct?

Yes because the DRBD storage is only available on the two nodes.

Basically you setup DRBD in primary/primary
Then you create an LVM volume on top of DRBD
Then in the GUI add that LVM storage:


That screen shot is of DRBD r0 on nodes vm5 and vm6
We named the LVM volume drbd0-vm5-vm6
The ID is the name I created in the Proxmox GUI vm5-vm6 so I know what nodes this storage is available on and the first node in the name, vm5, means the storage is used for VMS that primarily run on vm5.

DRBD r1, the 2nd DRBD volume is setup like this:

A question that intrigues me for a long time:
3- On the website:
http://pve.proxmox.com/wiki/Two-Node...bility_Cluster
literally says: "For this testing configuration, two DRBD resources were created, one for VM images an another one for VMs users data"

So, What is VMs users data?, what utility has? and how configure it?, because it not says more about it

Not really sure what is meant by users data.

 

[cesarpk]

Thank you very much e100, You are simply awesome.
I will do my tests today

And so soon I will test storage "NFS" on 2 PCs using "DRBD", for your great knowledge, you let me to ask you a few suggestions about it (apologizing for all the inconveniences).

In any case thank you very much , and God will bless you

Best regards
Cesar

 

[cesarpk]

Hi e100

One important message !!!:

With Proxmox VE 2.1 the lines 93 and 94 of the file /etc/lvm/lvm.conf said:
# You can turn off writing this cache file by setting this to 0.
write_cache_state = 1

The question:
if DRBD only copy blocks of disk, i believe that the configuration for LVM must be changed to 0
¿what is your opinion?

 

[e100]

I disabled the lvm write cache on my 1.X servers.
I have not done so on 2.X.

2.1 fixed the LVM related bugs so at this point I am assuming we will not need to make any mods to lvm.conf other than the filter line which is necessary when setting up DRBD.

 

[cesarpk]

2.1 fixed the LVM related bugs so at this point I am assuming we will not need to make any mods to lvm.conf other than the filter line which is necessary when setting up DRBD.

Excuse me e100, I'll just say my thinks in function to my common sense:
1- Proxmox and LVM don't know about of DRBD and what DRBD do (by being a separate layer).
2- As DRBD just copy what is written on the hard disk and not what is write in cache of LVM, i believe that disable LVM write cache is neccesary.
3- Proxmox or LVM do replica of cache of LVM on both nodes?... i don't think so

Why do you think different?. I missed of something?

Best regards
Cesar

 

[cesarpk]

Please udo, e100 or anybody that can help me !!!

As I have a cluster "Proxmox" (not Cluster HA) with only two nodes and without fence devices
And I had to reinstall Proxmox from scratch because I could not edit the file cluster.conf after having committed some errors.

Now, i need to create the file "cluster.conf.new" with the correct lines of configuration for not lose quorum due to the breakdown of a node.

So i need a file of example with his content complete, after i will follow the instructions as mentioned in the "how to" of Web Site of Proxmox based on the example file.

I was trying unsuccessfully to use a sample file I found on the website of redhat (and others websites):
http://docs.redhat.com/docs/en-US/Re...wo-node-cli-CA

i will be enormously grateful to anyone who can help me

Best regards
Cesar

 

[cesarpk]

Please udo, e100 or anybody that can help me !!!
Is urgent for me !!!

As I have a cluster "Proxmox" (not Cluster HA) with only two nodes and without fence devices
And I had to reinstall Proxmox from scratch because I could not edit the file cluster.conf after having committed some errors.

Now, i need to create the file "cluster.conf.new" with the correct lines of configuration for not lose quorum due to the breakdown of a node,and the live node don´t is disable !!!.


I was also trying activating "fencing devices" without success, Proxmox GUI not accept any of my settings.


So i need a example with content complete, after i will follow the instructions as mentioned in the "how to" of Web Site of Proxmox based on the example file.

I was trying unsuccessfully to use a sample file I found on the website of redhat (and others websites):
http://docs.redhat.com/docs/en-US/Re...wo-node-cli-CA
In proxmox website there is no an example with content complete, and doing juggle i got to run the "proxmox in cluster" (not cluster HA) if a node is turned off, after when I tried to add "manual fence devices," broke everything.
I suspect that if at first do things right, the final will be easier.

This is the current contents of my file cluster.conf:
<?xml version="1.0"?>
<cluster name="Cluster-Brain" config_version="2">

<cman keyfile="/var/lib/pve-cluster/corosync.authkey">
</cman>

<clusternodes>
<clusternode name="kvm1" votes="1" nodeid="1"/>
<clusternode name="kvm2" votes="1" nodeid="2"/></clusternodes>

</cluster>

i will be enormously grateful to anyone who can help me
Best regards
Cesar

 

[cesarpk]

Hi all
Please, anybody can help with little questions:

I am currently setting Proxmox with DRBDm and have 2 proxmox nodes (HA + manual fencing) with expected votes (work fine), and I want to configure "LV" correctly to boot differents VMs on both nodes (KVM and openvz for do snapshoots)
sdb on both nodes is for DRBD (for KVM and openvz VMs)

Questions:
should I configure only 1 PV and 1 VG with 2 LVs for sdb1?, or ...
2 PVs + 2 VGs + 2 LVs for sbd1 and sdb2 respectively?

Did any further requirement to make snapshoot for KVM and openvz?

Best regards
Cesar

 

[popovd.pro]

hi! it's seems look like

sdb1 -> drbdr0 (primary/primary) -> pv -> VGR0
sdb2 -> drbdr1 (primary/primary) -> pv -> VGR1

You not need to create LVs. LVs will creates when you add new VM. Just add both VGs to proxmox storage via WebConsole.

(Sorry for my English)

 

[cesarpk]

hi! it's seems look like

sdb1 -> drbdr0 (primary/primary) -> pv -> VGR0
sdb2 -> drbdr1 (primary/primary) -> pv -> VGR1

You not need to create LVs. LVs will creates when you add new VM. Just add both VGs to proxmox storage via WebConsole.

(Sorry for my English)

Thank you very much popovd.pro, I did as you recommended me and and it worked.
But I have some doubts:

1- I can not have KVM VMs with "qcow2" under "LVM Group", then would it be better to have a "LV" into "DRBD" and share this resource?
2- The question above is because if I need to extend the size of the VM disk is more complicated, or is there a quick way to do it with a few steps?
3- With Proxmox VE 2.1 and your suggestion, can i make live snapshot backup for KVM and OpenVZ? or do I need to configure something else?
4- And if I create a big LV for KVM and OpenVZ VMs into DRBD, can i make live snapshot backup for KVM and OpenVZ VMs?
5- What is your native language?. the mine is spanish if you want to speak me Spanish.

I would appreciate a lot if you can explain well.

Best regards
Cesar