just curious - you're running twelve nodes in six drbd-pairs, i assume? I wasn't aware that 2.0 can handle this; since for each two nodes the drbd device is the "shared storage", how does this harmonize with the other pairs? I thought storage could be marked as either "shared" or "local" for the whole cluster?
is there an alternative to fencing for HA?
- Thread starter hotwired007
- Start date
just curious - you're running twelve nodes in six drbd-pairs, i assume? I wasn't aware that 2.0 can handle this; since for each two nodes the drbd device is the "shared storage", how does this harmonize with the other pairs? I thought storage could be marked as either "shared" or "local" for the whole cluster?
Last edited:
cool, thanks for this info; this changes quite a bit for me - seems i should intensify testing a bit...
I am still confused about the hardware requirements for ProxMox HA.
Does all the nodes have to have the same exact hardware?
It would be nice if the ProxMox team come up with a hardware and software diagram to explain ProxMox HA for dummies like me. The diagram for ProxMox Mail Gateway HA was nicely done. It would be really great to see the same with ProxMox HA. The documentation doesn't do much justice.
Does all the nodes have to have the same exact hardware?
It would be nice if the ProxMox team come up with a hardware and software diagram to explain ProxMox HA for dummies like me. The diagram for ProxMox Mail Gateway HA was nicely done. It would be really great to see the same with ProxMox HA. The documentation doesn't do much justice.
Last edited:
It says: "This config was tested with DRAC V5 cards, but it must works with V6."
So it sounds like it works with DRAC5 since that is what it was tested against.
I Assume "but it must works with V6" is a typo that should say "but it should work with V6"
Thank you for clarifying. Really appreciate it.
1) How will that work if hardware does not need to be identical? Let say NODE A is running 16GB of RAM, NODE B w/ 8GB, NODE C w/ 10GB. If NODE A goes down and the VM require at least 12GB of RAM to function, how will ProxMox HA be supportive in this case?
2) Why do we need 3 nodes (servers)? I came from a Citrix Xensource and saw the HA setup can be done with 2 servers and a SAN.
3) How many NIC cards should I have on each of the servers?
4) If my understanding is correct, I only need enough hard drive space to install ProxMox and boot it up, then the VMs can be assigned to the NFS/SAN?
A picture is worth a thousand words. A diagram from ProxMox explaining HA (like how it was done with ProxMox Mail Gateway HA) would be splendid!
Thank you for your help.
Hardware does not need to be identical, but you need to make sure there are enought ressources (RAM).
You can also do such setup with PVE (use the SAN as quorum disk).
at least 2
yes
Thank you very much.
A) I am still unclear as to why Proxmox stated we need 3 nodes for HA. Based on your suggestion, Proxmox HA would run fine on 2 nodes. This is really important to me as each server run close to about 14K-16K each. Does the third node really necessary? What is the purpose of having 3 nodes when HA will work find on 2 nodes?
B) Please also help me to better understand Proxmox HA. Can it be use as a cloud environment? Let say I have 3 nodes. Do I place all the VM on nodes 1 and let node 2 and 3 take over in the event of the failure? In this setup, node 2 and 3's hardware resource are somewhat wasted. They are just standing by waiting for a failure and do nothing else.
I would like to utilize Proxmox HA like a cloud environment where all the hardware are being utilize to its maximum potential, as well as being able to serve as fail-over. Is this possible? Sorry for the newbie questions.
C) Could you please clarify the need for each server to have 2 NICs? How would be they be connected?
You need at least two nodes.
It is suggested to have three.
If you have two nodes A and B, something happens and A and B can not talk to each other. Who/what is the problem? Is node A dead? Is node B dead? there is no majority to decide a proper course of action so all actions are based on assumptions.
If you have three nodes A, B and C, something is wrong with A. A can not talk to B and C. B and C can talk to each other but not A. B and C, the majority of nodes(quorum) agree that A is not responding and can fence A to prevent it from causing problems.
My post above explains this in greater detail, if you missed it:http://forum.proxmox.com/threads/8693-is-there-an-alternative-to-fencing-for-HA?p=49323#post49323
We are using a really cheap old server to act as the third node. It does not run any VMs, It just participates in the cluster as a quorum member.
Once we add more real nodes we will remove the cheap old server as it will no longer be needed.
Thank you very much! Your explanation is very clear. I have more questions if you don't mind:
- So in the current setup, node C as an old server being use as a communication node for the cluster? if node A and B happened to fail, node C will not do anything since the VMs are not on it?
- Do you load all the VMs on node A and leave node B empty, or do you load VMs on both?
- I am trying to figure out what would happen if VMs are loaded on both. Let say node B failed, how would node A be able to handle the resources for its own VMs plus taking on the VMs from node B?
Node C is just there to be a witness to what is going on in the cluster to provide proper quorum.
If both Node A and B fail at the same time then no VMs run.
In my setup I used DRBD to replicate data between Node A and B.
As suggested in the wiki I have two DRBD volumes (DRBD0 and DRBD1) NodeA runs the VMs stored on DRBD0 and Node B runs the VMs on DRBD1.
If Node A fails then node B runs all the VMs.
If Node B fails then node A runs all the VMs.
I am using Failover Domains for HA VMs, this is not yet working in the GUI but gives you more control over what node the HA VMs will run on.
We try to keep the resource usage on a single node to around 50%-60%.
Ie we only assign around 60% of the RAM on nodeA and on nodeB to VMs.
This way if a node fails, the other node can run everything (once KSM kicks in)
Sometimes we have VMs that are redundant(ie redundant web server VMs, memcached servers), where if one was not running things keep working.
We run those on a node to utilise the 40%-50% idle resources.
Then during a failure simply turn that VM off so we can run the most important VMs on the single node.
You must consider your requirements and available resources when setting things up for HA.
Part of that planning is ensuring that all the important stuff needed to stay online can run on one node at the level of performance that is required.
may i ask - how did you 'flag' that 3rd 'node C' to be sort of dummy/quorum-only?
What i see in a 2-node-HA-setup-with-3rd-as-quorum is, that in case 'node A' fails, the cluster (randomly?) tries to make "node C" taking over the VMs. Though this quite immediately fails, and the correct 'node B' then finally takes over afterwards, this introduces an additional latency, and imo also an additional amount of 'risk'...
The dummy 'node C' does not have "rights" on the shared storage, where VMs are running on - i hoped that would be that mentioned "flag", but it seems not to be the case this way...
i think he has a 2-node DRBD setup, as he said before
so the vms can be managed only by nodes accessing those shared volumes...
node C can "vote", as it is part of the pve cluster, but not run the vms because it is not part of the "drbd cluster"
Marco
Code:
In my setup I used DRBD to replicate data between Node A and B.
As suggested in the wiki I have two DRBD volumes (DRBD0 and DRBD1) NodeA runs the VMs stored on DRBD0 and Node B runs the VMs on DRBD1.
If Node A fails then node B runs all the VMs.
If Node B fails then node A runs all the VMs.so the vms can be managed only by nodes accessing those shared volumes...
node C can "vote", as it is part of the pve cluster, but not run the vms because it is not part of the "drbd cluster"
Marco
In the cluster.conf:
Code:
<rm>
<pvevm autostart="1" vmid="101" domain="nodeA-nodeB"/>
<failoverdomains>
<failoverdomain name="nodeA-nodeB" restricted="1" ordered="1" nofailback="1">
<failoverdomainnode name="nodeA" priority="1"/>
<failoverdomainnode name="nodeB" priority="100"/>
</failoverdomain>
</failoverdomains>
</rm>