"Best practise" for a two node proxmox/storage cluster

[sherminator]

Today I did my first crash test ;-) I will describe my results tomorrow.

So with a little delay I like to describe what I tried as my first crash test:

I populated my nodes like this:
node1: VM1 and VM2
node2: VM3 and VM4

Everything works fine, live migration is no problem.
Now I turned off node1 - no gentle shutdown, rather the hard way: Press and hold the power button via ILO.
So what happened? Of course, node1, VM1 and VM2 went off immediately.
node2, VM3 and VM4 kept on running.
Then I logged in into node2 via SSH and moved all VM configs from node1 to node2:

# mv /etc/pve/nodes/node1/qemu-server/* /etc/pve/nodes/node2/qemu-server/

Thereby VM1 and VM2 appeared in the node2 section inside the WebGUI. Finally I started these VMs on node2 so that all VMs run on this node.

That's what I wanted - business can go on and I can repair node1.

After I had finished "repairing" node1 I turned it on again. DRBD started to replicate as expected, and after a short while everything was in sync again. I migrated VM3 and VM4 back to node1 and everything looked like before.

From my point of view this was a success, what do you think?

Greets
Stephan

 

[sherminator]

it is entirely possible that only the remote dataset contains the correct data on a local disk fault, controller fault, memory, etc; This page may explain it better. If you want an asynchronous method, you may want to use zfs + zfs send.

Are you sure? In my understanding of DRBD in primary/secondary mode data is written to the primary device first, then replicated to the secondary. If this is true, primary would always be newer than secondary.

laughs.

What's so funny about this?

To wit- DRBD with 2 nodes is likely to cause you far more pain then a single iSCSI NAS and two hypervisors. painful experience speaking.

We run a two node DRBD cluster for the last two years without a single issue. By the way: Our setup is designed and administrated by Linbit, so I strongly believe that it can't be so bad. ;-)

Greets
Stephan

 

[alexskysilk]

Are you sure? In my understanding of DRBD in primary/secondary mode data is written to the primary device first, then replicated to the secondary. If this is true, primary would always be newer than secondary.

Its not important if I'm sure. One of us is right. one of us is building his cluster based on this assumption. If your intention is to convince me you're right, we can close the thread- it doesnt matter to me.

What's so funny about this?

because its obvious. I'm not laughing out of mirth, just amusement.

We run a two node DRBD cluster for the last two years without a single issue. By the way: Our setup is designed and administrated by Linbit, so I strongly believe that it can't be so bad. ;-)

Everything works, until it doesnt. In any event, I wish you the best of luck; its entirely conceivable you'll be able to run your application without trouble for the entire life of the project. other people have. best practices ≠ must follow.

 

[alexskysilk]

Apparently it works perfectly so I don't undertand why people say two nodes are bad.

Not bad at all. this is a normal, well supported setup and you can migrate VMs from one hypervisor to the other; its just not an HA cluster.

 

[dcsapak]

From my point of view this was a success, what do you think?

you could try the following (just for additional testing)
assuming you have a single switch/cable connecting your nodes:

shut it off/remove it, wait a minute, turn it back on/plug it back in

what does happen then?

a hard node shutdown is one of the easier failures to recover from, an intermittent network error, not so much

 

[sherminator]

Its not important if I'm sure. One of us is right. one of us is building his cluster based on this assumption. If your intention is to convince me you're right, we can close the thread- it doesnt matter to me.

For me it's neither about believing nor convincing. It's about understanding the technology so that I can evaluate the risks. This is what I'm trying to do in this thread - and I'm very glad about everyone who participates!

Greets
Stephan

 

[sherminator]

Hi Dominik,

you could try the following (just for additional testing)
assuming you have a single switch/cable connecting your nodes:

thanks a lot for your testing suggestion!
Unfortunately these two nodes are connected to productive switches, so I can't turn the switches off. Furthermore the nodes are located in two different buildings, so I can't plug off all network cables at once.
What I tried is to plug the two network cables ("VM network" and "DRBD network") off node1. Everything behaved like expected: node1, VM1 and VM2 kept on running, but weren't reachable. node2, VM3 and VM4 were reachable all the time. Then I plugged the cables in again. node1, VM1 and VM2 reappeared, DRBD synched and everthing looked good again.

To be closer to your suggestion, next I will try to turn all four switchports off at once - maybe that will more look like a switch crash.

Many greets
Stephan

 

[Imilah]

Hai, sherminator

Have you check your data when you do a manual migration (when the node 1 die, and you turn on VMs in node 2) ? I do success (I think) on doing your scenario, but I haven't check if no data is missing. Have you check it?